| Message ID | 20240607142423.116285-2-pvorel@suse.cz |
|---|---|
| State | Accepted |
| Headers | show |
| Series | Deprecate runltp (please use kirk) | expand |
On Fri, 2024-06-07 at 16:24 +0200, Petr Vorel via lists.yoctoproject.org wrote: > kirk is not perfect but already much better than old runltp script. > Let's deprecate runltp and propagate kirk. > > Signed-off-by: Petr Vorel <pvorel@suse.cz> > --- > runltp | 13 +++++++++++++ > 1 file changed, 13 insertions(+) I'd note that Yocto Project's CI is still using runltp and we have no recipe for kirk, or any experience of using it. This does therefore worry me a little bit, there appears to be a lot of complexity in kirk we don't need. Cheers, Richard
Hi! > > kirk is not perfect but already much better than old runltp script. > > Let's deprecate runltp and propagate kirk. > > > > Signed-off-by: Petr Vorel <pvorel@suse.cz> > > --- > > runltp | 13 +++++++++++++ > > 1 file changed, 13 insertions(+) > > I'd note that Yocto Project's CI is still using runltp and we have no > recipe for kirk, or any experience of using it. That's why runltp isn't going to disappear without a deprecation period, the idea is to add the deprecation and wait a few releases before the final removal, that in practical terms means at least a year, possibly two for users to explore the replacement and give us feedback. > This does therefore worry me a little bit, there appears to be a lot > of complexity in kirk we don't need. I would say that there is a complexity that you do not think that you need but in reality you do. First of all the assumption that you can have the test runner that keeps the results and overall state on the same machine that runs the tests is the most flawed of them all. So running the tests over some kind of connection is the basis design principle of kirk. That allows us to easily and safely detect when we crash kernels with our tests, which tend to happen more often than most people think. And I can go about all the things that are there because of a good reasons for hours. That being said, the current kirk implementation ended up more complex than I would like it, and that is something to improve over the deprecation period. The general idea is to allow users to experiment with kirk, even when it's not perfect to get feedback and ideally make it usable for most usecases before we get rid of runltp for good.
On Fri, 2024-06-07 at 17:45 +0200, Cyril Hrubis wrote: > Hi! > > > kirk is not perfect but already much better than old runltp script. > > > Let's deprecate runltp and propagate kirk. > > > > > > Signed-off-by: Petr Vorel <pvorel@suse.cz> > > > --- > > > runltp | 13 +++++++++++++ > > > 1 file changed, 13 insertions(+) > > > > I'd note that Yocto Project's CI is still using runltp and we have no > > recipe for kirk, or any experience of using it. > > That's why runltp isn't going to disappear without a deprecation period, > the idea is to add the deprecation and wait a few releases before the > final removal, that in practical terms means at least a year, possibly > two for users to explore the replacement and give us feedback. It helps to know that, it wasn't clear how quickly you planed to remove runltp! > > This does therefore worry me a little bit, there appears to be a lot > > of complexity in kirk we don't need. > > I would say that there is a complexity that you do not think that you > need but in reality you do. First of all the assumption that you can > have the test runner that keeps the results and overall state on the > same machine that runs the tests is the most flawed of them all. So > running the tests over some kind of connection is the basis design > principle of kirk. That allows us to easily and safely detect when we > crash kernels with our tests, which tend to happen more often than most > people think. And I can go about all the things that are there because > of a good reasons for hours. I think you misunderstand my point. Yocto project already has code to handle setting up qemu instances, connecting to them, collecting data from them etc. and we use that with ltp in the same way we use it for lots of other tests. So yes, I agree with you that you need a connection but we already have a solution for that. We probably don't want some tests doing this with kirk and everything else doing it differently. I suspect we wouldn't want to switch everything we're doing over to kirk either as that wouldn't work for us or the kirk maintainers due to differing needs and expectations. > That being said, the current kirk implementation ended up more complex > than I would like it, and that is something to improve over the > deprecation period. The general idea is to allow users to experiment > with kirk, even when it's not perfect to get feedback and ideally make > it usable for most usecases before we get rid of runltp for good. It sounds like we need to switch to kirk and use it simply as a direct run host driver, but we are going to have a lot of complexity in there we aren't in need of. Cheers, Richard
> -----Original Message----- > From: Richard Purdie <richard.purdie@linuxfoundation.org> > On Fri, 2024-06-07 at 17:45 +0200, Cyril Hrubis wrote: > > Hi! > > > > kirk is not perfect but already much better than old runltp script. > > > > Let's deprecate runltp and propagate kirk. > > > > > > > > Signed-off-by: Petr Vorel <pvorel@suse.cz> > > > > --- > > > > runltp | 13 +++++++++++++ > > > > 1 file changed, 13 insertions(+) > > > > > > I'd note that Yocto Project's CI is still using runltp and we have no > > > recipe for kirk, or any experience of using it. > > > > That's why runltp isn't going to disappear without a deprecation period, > > the idea is to add the deprecation and wait a few releases before the > > final removal, that in practical terms means at least a year, possibly > > two for users to explore the replacement and give us feedback. > > It helps to know that, it wasn't clear how quickly you planed to remove > runltp! > > > > This does therefore worry me a little bit, there appears to be a lot > > > of complexity in kirk we don't need. > > > > I would say that there is a complexity that you do not think that you > > need but in reality you do. First of all the assumption that you can > > have the test runner that keeps the results and overall state on the > > same machine that runs the tests is the most flawed of them all. So > > running the tests over some kind of connection is the basis design > > principle of kirk. That allows us to easily and safely detect when we > > crash kernels with our tests, which tend to happen more often than most > > people think. And I can go about all the things that are there because > > of a good reasons for hours. > > I think you misunderstand my point. Yocto project already has code to > handle setting up qemu instances, connecting to them, collecting data > from them etc. and we use that with ltp in the same way we use it for > lots of other tests. So yes, I agree with you that you need a > connection but we already have a solution for that. > > We probably don't want some tests doing this with kirk and everything > else doing it differently. I suspect we wouldn't want to switch > everything we're doing over to kirk either as that wouldn't work for us > or the kirk maintainers due to differing needs and expectations. > > > That being said, the current kirk implementation ended up more complex > > than I would like it, and that is something to improve over the > > deprecation period. The general idea is to allow users to experiment > > with kirk, even when it's not perfect to get feedback and ideally make > > it usable for most usecases before we get rid of runltp for good. > > It sounds like we need to switch to kirk and use it simply as a direct > run host driver, but we are going to have a lot of complexity in there > we aren't in need of. I'm in the same position as Richard here. Fuego uses runltp on the target (device under test), and has it's own mechanisms for detecting timeouts or kernel crashes, gathering test output, restarting targets, etc. from a test host. These same mechanisms are used for other tests. Fuego also has mechanisms for running individual LTP tests on the device (by installing the individual test, executing it, gathering results remotely, and removing the test and test artifacts). This is used in cases where the overhead of installing and running runltp is too big. I haven't investigated kirk yet (but it was on my list of things to do). Will kirk and/or LTP provide a simple mechanism for quick install and execution of individual tests or small sets of tests (and itself)? Fuego's model of testing is geared towards testing of production devices, where no test artifacts are pre-installed on the target, and full cleanup (removal of tests and test artifacts) is done between tests. -- Tim
Hi! > > > This does therefore worry me a little bit, there appears to be a lot > > > of complexity in kirk we don't need. > > > > I would say that there is a complexity that you do not think that you > > need but in reality you do. First of all the assumption that you can > > have the test runner that keeps the results and overall state on the > > same machine that runs the tests is the most flawed of them all. So > > running the tests over some kind of connection is the basis design > > principle of kirk. That allows us to easily and safely detect when we > > crash kernels with our tests, which tend to happen more often than most > > people think. And I can go about all the things that are there because > > of a good reasons for hours. > > I think you misunderstand my point. Yocto project already has code to > handle setting up qemu instances, connecting to them, collecting data > from them etc. and we use that with ltp in the same way we use it for > lots of other tests. So yes, I agree with you that you need a > connection but we already have a solution for that. > > We probably don't want some tests doing this with kirk and everything > else doing it differently. I suspect we wouldn't want to switch > everything we're doing over to kirk either as that wouldn't work for us > or the kirk maintainers due to differing needs and expectations. This may actually work, since we are trying to make kirk flexible enough to run other testsuites, I think that we already run subset of selftest with kirk in our environment. > > That being said, the current kirk implementation ended up more complex > > than I would like it, and that is something to improve over the > > deprecation period. The general idea is to allow users to experiment > > with kirk, even when it's not perfect to get feedback and ideally make > > it usable for most usecases before we get rid of runltp for good. > > It sounds like we need to switch to kirk and use it simply as a direct > run host driver, but we are going to have a lot of complexity in there > we aren't in need of. I'm afraid that's not a good solution either. The end goal for kirk is to have a small binary locked in RAM and with realtime priority to execute tests and send back logs, in case of qemu over virtio, to the kirk. That is to make sure that logs are collected properly even when kernel is out of memory and in a similar situations. If you run kirk on the VM, reporting is not going to be reliable. My initial idea was to build the new generation testrunner as a set of building blocks, that could be reused separately, but the current desing does not make it easy. We do have the ltx binary, which is the small dispatcher supposed to run on the VM. And in an ideal world we would have a python library that talks to it on the other end, as a part of kirk, that could be reused separately. And the same for building lists of test to execute, ideally we would have a python library that would export a simple interface so that everyone could integrate the blocks that they really need into their solution.
Hi! > I'm in the same position as Richard here. Fuego uses runltp on the target (device under test), > and has it's own mechanisms for detecting timeouts or kernel crashes, gathering test output, > restarting targets, etc. from a test host. These same mechanisms are used for other tests. > Fuego also has mechanisms for running individual LTP tests on the device (by installing the > individual test, executing it, gathering results remotely, and removing the test and test artifacts). > This is used in cases where the overhead of installing and running runltp is too big. I haven't > investigated kirk yet (but it was on my list of things to do). If you are executing a single test you do not need runltp. The current added value of runltp is that it loops over all tests in a set of runtest files, collects the results and saves overall status. If you want to run a single test, you need to set up a few environment variables and you are good to go. > Will kirk and/or LTP provide a simple mechanism for quick install and execution of > individual tests or small sets of tests (and itself)? Fuego's model of testing is geared towards > testing of production devices, where no test artifacts are pre-installed on the target, > and full cleanup (removal of tests and test artifacts) is done between tests. Currently installation is completely out of scope for kirk. The workflow we have expects LTP installed and first step kirk does is to download runtest files and metadata from the device under test. And I'm perfectly aware that single workflow will not fit everybody, that's why my initial idea was a set of reusable components rather than a full solution.
On Fri, 2024-06-07 at 18:35 +0200, Cyril Hrubis wrote: > Hi! > > > > This does therefore worry me a little bit, there appears to be a lot > > > > of complexity in kirk we don't need. > > > > > > I would say that there is a complexity that you do not think that you > > > need but in reality you do. First of all the assumption that you can > > > have the test runner that keeps the results and overall state on the > > > same machine that runs the tests is the most flawed of them all. So > > > running the tests over some kind of connection is the basis design > > > principle of kirk. That allows us to easily and safely detect when we > > > crash kernels with our tests, which tend to happen more often than most > > > people think. And I can go about all the things that are there because > > > of a good reasons for hours. > > > > I think you misunderstand my point. Yocto project already has code to > > handle setting up qemu instances, connecting to them, collecting data > > from them etc. and we use that with ltp in the same way we use it for > > lots of other tests. So yes, I agree with you that you need a > > connection but we already have a solution for that. > > > > We probably don't want some tests doing this with kirk and everything > > else doing it differently. I suspect we wouldn't want to switch > > everything we're doing over to kirk either as that wouldn't work for us > > or the kirk maintainers due to differing needs and expectations. > > This may actually work, since we are trying to make kirk flexible enough > to run other testsuites, I think that we already run subset of selftest > with kirk in our environment. I'm not convinced that would be a great fit for either project to be honest. Reading more below, you have a very specific idea of how communication should happen and many of our test workflow needs are going to be outside of scope of a single binary communicating over virtio. It makes sense for kernel testing but we have other needs. > > > That being said, the current kirk implementation ended up more complex > > > than I would like it, and that is something to improve over the > > > deprecation period. The general idea is to allow users to experiment > > > with kirk, even when it's not perfect to get feedback and ideally make > > > it usable for most usecases before we get rid of runltp for good. > > > > It sounds like we need to switch to kirk and use it simply as a direct > > run host driver, but we are going to have a lot of complexity in there > > we aren't in need of. > > I'm afraid that's not a good solution either. The end goal for kirk is > to have a small binary locked in RAM and with realtime priority to > execute tests and send back logs, in case of qemu over virtio, to the > kirk. That is to make sure that logs are collected properly even when > kernel is out of memory and in a similar situations. > > If you run kirk on the VM, reporting is not going to be reliable. This means you're effectively mandating how ltp be run and the only variable would be the kernel binary. Whilst I can understand that, I'm not sure how useful us testing with this would be. > My initial idea was to build the new generation testrunner as a set of > building blocks, that could be reused separately, but the current desing > does not make it easy. > > We do have the ltx binary, which is the small dispatcher supposed to run > on the VM. And in an ideal world we would have a python library that > talks to it on the other end, as a part of kirk, that could be reused > separately. And the same for building lists of test to execute, ideally > we would have a python library that would export a simple interface so > that everyone could integrate the blocks that they really need into > their solution. Automated testing is a hard problem to solve generically and even if you do manage that, this all looks like a lot of work even just to reproduce what works today :/. I do understand the idea but in practise, I don't have the time or know anyone with the time to put into something like this. I can barely keep the tests/infra we have running. Cheers, Richard
Hi! > > This may actually work, since we are trying to make kirk flexible enough > > to run other testsuites, I think that we already run subset of selftest > > with kirk in our environment. > > I'm not convinced that would be a great fit for either project to be > honest. Reading more below, you have a very specific idea of how > communication should happen and many of our test workflow needs are > going to be outside of scope of a single binary communicating over > virtio. It makes sense for kernel testing but we have other needs. Fair enough. > > > > That being said, the current kirk implementation ended up more complex > > > > than I would like it, and that is something to improve over the > > > > deprecation period. The general idea is to allow users to experiment > > > > with kirk, even when it's not perfect to get feedback and ideally make > > > > it usable for most usecases before we get rid of runltp for good. > > > > > > It sounds like we need to switch to kirk and use it simply as a direct > > > run host driver, but we are going to have a lot of complexity in there > > > we aren't in need of. > > > > I'm afraid that's not a good solution either. The end goal for kirk is > > to have a small binary locked in RAM and with realtime priority to > > execute tests and send back logs, in case of qemu over virtio, to the > > kirk. That is to make sure that logs are collected properly even when > > kernel is out of memory and in a similar situations. > > > > If you run kirk on the VM, reporting is not going to be reliable. > > This means you're effectively mandating how ltp be run and the only > variable would be the kernel binary. Whilst I can understand that, I'm > not sure how useful us testing with this would be. Not at all. As I replied to Tim, there is no secret sauce in runltp or kirk. In the end it's a tool to execute a test binaries. If you have a system that can execute binaries, reliably transfer logs and handle kernel crashes you can as well just execute the tests yourself. All you need from us is a tooling that will produce a list(s) of tests to execute. > > My initial idea was to build the new generation testrunner as a set of > > building blocks, that could be reused separately, but the current desing > > does not make it easy. > > > > We do have the ltx binary, which is the small dispatcher supposed to run > > on the VM. And in an ideal world we would have a python library that > > talks to it on the other end, as a part of kirk, that could be reused > > separately. And the same for building lists of test to execute, ideally > > we would have a python library that would export a simple interface so > > that everyone could integrate the blocks that they really need into > > their solution. > > Automated testing is a hard problem to solve generically and even if > you do manage that, this all looks like a lot of work even just to > reproduce what works today :/. Indeed. However I stil think that there are reusable parts that may be worth putting together. > I do understand the idea but in practise, I don't have the time or know > anyone with the time to put into something like this. I can barely keep > the tests/infra we have running. This reminds me: https://en.wikipedia.org/wiki/Boots_theory If we apply that to software not being able to inovate keeps you doing work that shouldn't needed to be done in the first place.
> -----Original Message----- > From: Cyril Hrubis <chrubis@suse.cz> > > > I'm afraid that's not a good solution either. The end goal for kirk is > > > to have a small binary locked in RAM and with realtime priority to > > > execute tests and send back logs, in case of qemu over virtio, to the > > > kirk. That is to make sure that logs are collected properly even when > > > kernel is out of memory and in a similar situations. > > > > > > If you run kirk on the VM, reporting is not going to be reliable. > > > > This means you're effectively mandating how ltp be run and the only > > variable would be the kernel binary. Whilst I can understand that, I'm > > not sure how useful us testing with this would be. > > Not at all. As I replied to Tim, there is no secret sauce in runltp or > kirk. In the end it's a tool to execute a test binaries. If you have a > system that can execute binaries, reliably transfer logs and handle > kernel crashes you can as well just execute the tests yourself. All you > need from us is a tooling that will produce a list(s) of tests to > execute. I don't think it's that simple. Currently, Fuego users can choose to either: 1. run a suite of tests (specified in the runtest file) using runltp executing on the target 2. run an individual test, not using runltp. In the first case, since some of the suites have a large number of tests, there are options in Fuego to convert the results into spreadsheet files or PDF reports. But this is based on the multi-test output from runltp. Does kirk provide the same output formats and output options as runltp? If runltp is eventually removed, I'll have to come up with a solution for executing suites of tests on the target, and making sure the output is the same as runltp (or modifying the report generation code to handle a new output). Fuego supports multiple "transport" layers. ssh, serial console, and adb transfers are supported, as well as a few weird transports (such as ssh to a controller board that then transfers over serial). I wouldn't request kirk to support these oddball transport mechanisms, but if it had support for mapping it's communication mechanisms onto these, or supporting some pluggable mechanism for transferring files and executing programs (collecting stdout, stderr and return code), that might be useful. ... > > > > > > We do have the ltx binary, which is the small dispatcher supposed to run > > > on the VM. And in an ideal world we would have a python library that > > > talks to it on the other end, as a part of kirk, that could be reused > > > separately. And the same for building lists of test to execute, ideally > > > we would have a python library that would export a simple interface so > > > that everyone could integrate the blocks that they really need into > > > their solution. > > > > Automated testing is a hard problem to solve generically and even if > > you do manage that, this all looks like a lot of work even just to > > reproduce what works today :/. > > Indeed. However I stil think that there are reusable parts that may be > worth putting together. Possibly. I tried for a few years to integrate Fuego and KernelCI, but their architectures were too different, and I eventually gave up. Now that KernelCI is changing, I've thought about going back and seeing if I could transfer anything between the projects. Usually, you have to write some kind of mapping layer, and the mappings turn out to be harder than expected, due to assumptions baked in to the architecture. -- Tim
On Fri, Jun 7, 2024 at 7:50 AM Richard Purdie via lists.yoctoproject.org <richard.purdie=linuxfoundation.org@lists.yoctoproject.org> wrote: > On Fri, 2024-06-07 at 16:24 +0200, Petr Vorel via > lists.yoctoproject.org wrote: > > kirk is not perfect but already much better than old runltp script. > > Let's deprecate runltp and propagate kirk. > > > > Signed-off-by: Petr Vorel <pvorel@suse.cz> > > --- > > runltp | 13 +++++++++++++ > > 1 file changed, 13 insertions(+) > > I'd note that Yocto Project's CI is still using runltp and we have no > recipe for kirk, or any experience of using it. This does therefore > worry me a little bit, there appears to be a lot of complexity in kirk > we don't need. > I have begun a recipe for kirk. I know very little about the rest of how we (Yocto Project) run ltp or how kirk is meant to be used. https://git.yoctoproject.org/poky-contrib/log/?h=timo/kirk > > Cheers, > > Richard > > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#1302): > https://lists.yoctoproject.org/g/automated-testing/message/1302 > Mute This Topic: https://lists.yoctoproject.org/mt/106543727/924729 > Group Owner: automated-testing+owner@lists.yoctoproject.org > Unsubscribe: https://lists.yoctoproject.org/g/automated-testing/unsub [ > ticotimo@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- > >
Hi! My 2 cents below. On 6/7/24 23:17, Bird, Tim wrote: >> -----Original Message----- >> From: Cyril Hrubis <chrubis@suse.cz> >>>> I'm afraid that's not a good solution either. The end goal for kirk is >>>> to have a small binary locked in RAM and with realtime priority to >>>> execute tests and send back logs, in case of qemu over virtio, to the >>>> kirk. That is to make sure that logs are collected properly even when >>>> kernel is out of memory and in a similar situations. >>>> >>>> If you run kirk on the VM, reporting is not going to be reliable. >>> This means you're effectively mandating how ltp be run and the only >>> variable would be the kernel binary. Whilst I can understand that, I'm >>> not sure how useful us testing with this would be. >> Not at all. As I replied to Tim, there is no secret sauce in runltp or >> kirk. In the end it's a tool to execute a test binaries. If you have a >> system that can execute binaries, reliably transfer logs and handle >> kernel crashes you can as well just execute the tests yourself. All you >> need from us is a tooling that will produce a list(s) of tests to >> execute. > I don't think it's that simple. > > Currently, Fuego users can choose to either: > 1. run a suite of tests (specified in the runtest file) using runltp executing on the target > 2. run an individual test, not using runltp. > > In the first case, since some of the suites have a large number of tests, > there are options in Fuego to convert the results into spreadsheet files > or PDF reports. But this is based on the multi-test output from runltp. > > Does kirk provide the same output formats and output options as runltp? No, but it provides something better than that: a JSON file that can be easily parsed by most of the libraries out there. kirk can be literally used as it is runltp, with difference of python dependency. If you have python, you can run kirk on host as well as on target. All features like Qemu, LTX, SSH (etc.) supports are meant to be used by host only. > > If runltp is eventually removed, I'll have to come up with a solution > for executing suites of tests on the target, and making sure the output > is the same as runltp (or modifying the report generation code to handle > a new output). > > Fuego supports multiple "transport" layers. ssh, serial console, > and adb transfers are supported, as well as a few weird transports > (such as ssh to a controller board that then transfers over serial). > I wouldn't request kirk to support these oddball transport mechanisms, > but if it had support for mapping it's communication mechanisms > onto these, or supporting some pluggable mechanism for transferring > files and executing programs (collecting stdout, stderr and return code), > that might be useful. > > ... >>>> We do have the ltx binary, which is the small dispatcher supposed to run >>>> on the VM. And in an ideal world we would have a python library that >>>> talks to it on the other end, as a part of kirk, that could be reused >>>> separately. And the same for building lists of test to execute, ideally >>>> we would have a python library that would export a simple interface so >>>> that everyone could integrate the blocks that they really need into >>>> their solution. >>> Automated testing is a hard problem to solve generically and even if >>> you do manage that, this all looks like a lot of work even just to >>> reproduce what works today :/. >> Indeed. However I stil think that there are reusable parts that may be >> worth putting together. > Possibly. I tried for a few years to integrate Fuego and KernelCI, > but their architectures were too different, and I eventually gave up. > Now that KernelCI is changing, I've thought about going back > and seeing if I could transfer anything between the projects. > > Usually, you have to write some kind of mapping layer, and the mappings > turn out to be harder than expected, due to assumptions baked in to > the architecture. > > -- Tim > > Regards, Andrea Cervesato
Hi! > I don't think it's that simple. > > Currently, Fuego users can choose to either: > 1. run a suite of tests (specified in the runtest file) using runltp executing on the target And this is something that I think is a suboptimal solution. If the run crashes in the middle you can't easily continue with the rest of the tests after the system under test has been rebooted. So the desing decision for kirk was that the complete state of the testsuite execution is being kept on the machine that drives the testing. I did a talk at SUSE Labs conference when I tried to explain when we are heading and it was recorded, unfortunately it wasn't uploaded to youtube yet, I will post the link here once it's there. The bottom line question is how much code from us can be reused in your environment and this is something we have to figure out. > 2. run an individual test, not using runltp. > > In the first case, since some of the suites have a large number of tests, > there are options in Fuego to convert the results into spreadsheet files > or PDF reports. But this is based on the multi-test output from runltp. > > Does kirk provide the same output formats and output options as runltp? We decided to go for JSON format for generic the test results for kirk. However given that we store the results in python data structures internally, it's easy to write an exporter for a format you want and produce that directly. We can add CSV exporter for spreadsheets in 100 lines of python. The JSON exported looks like this: https://github.com/linux-test-project/kirk/blob/master/libkirk/export.py > If runltp is eventually removed, I'll have to come up with a solution > for executing suites of tests on the target, and making sure the output > is the same as runltp (or modifying the report generation code to handle > a new output). The second one is preferable. > Fuego supports multiple "transport" layers. ssh, serial console, > and adb transfers are supported, as well as a few weird transports > (such as ssh to a controller board that then transfers over serial). > I wouldn't request kirk to support these oddball transport mechanisms, > but if it had support for mapping it's communication mechanisms > onto these, or supporting some pluggable mechanism for transferring > files and executing programs (collecting stdout, stderr and return code), > that might be useful. That is actually the end goal. The ltx is designed to marshal messages over a byte oriented connection, the supported commands at the moment can execute binaries, transfer files, etc. We intend to use it over virtio and physical serial ports as well. So the idea is that once ltx is started on the system uder test kirk connect to that instance to drive the testing. It's not production ready yet, and that is something that I have on my TODO. > > > > We do have the ltx binary, which is the small dispatcher supposed to run > > > > on the VM. And in an ideal world we would have a python library that > > > > talks to it on the other end, as a part of kirk, that could be reused > > > > separately. And the same for building lists of test to execute, ideally > > > > we would have a python library that would export a simple interface so > > > > that everyone could integrate the blocks that they really need into > > > > their solution. > > > > > > Automated testing is a hard problem to solve generically and even if > > > you do manage that, this all looks like a lot of work even just to > > > reproduce what works today :/. > > > > Indeed. However I stil think that there are reusable parts that may be > > worth putting together. > > Possibly. I tried for a few years to integrate Fuego and KernelCI, > but their architectures were too different, and I eventually gave up. > Now that KernelCI is changing, I've thought about going back > and seeing if I could transfer anything between the projects. > > Usually, you have to write some kind of mapping layer, and the mappings > turn out to be harder than expected, due to assumptions baked in to > the architecture. That's what I'm hoping to avoid to keep things in a simple modules, but I guess that we will run into a lot of implementation details that will make things hard or even impossible...
> -----Original Message----- > From: ltp <ltp-bounces+tim.bird=sony.com@lists.linux.it> On Behalf Of Andrea Cervesato via ltp > Hi! > > My 2 cents below. > > On 6/7/24 23:17, Bird, Tim wrote: > >> -----Original Message----- > >> From: Cyril Hrubis <chrubis@suse.cz> > >>>> I'm afraid that's not a good solution either. The end goal for kirk is > >>>> to have a small binary locked in RAM and with realtime priority to > >>>> execute tests and send back logs, in case of qemu over virtio, to the > >>>> kirk. That is to make sure that logs are collected properly even when > >>>> kernel is out of memory and in a similar situations. > >>>> > >>>> If you run kirk on the VM, reporting is not going to be reliable. > >>> This means you're effectively mandating how ltp be run and the only > >>> variable would be the kernel binary. Whilst I can understand that, I'm > >>> not sure how useful us testing with this would be. > >> Not at all. As I replied to Tim, there is no secret sauce in runltp or > >> kirk. In the end it's a tool to execute a test binaries. If you have a > >> system that can execute binaries, reliably transfer logs and handle > >> kernel crashes you can as well just execute the tests yourself. All you > >> need from us is a tooling that will produce a list(s) of tests to > >> execute. > > I don't think it's that simple. > > > > Currently, Fuego users can choose to either: > > 1. run a suite of tests (specified in the runtest file) using runltp executing on the target > > 2. run an individual test, not using runltp. > > > > In the first case, since some of the suites have a large number of tests, > > there are options in Fuego to convert the results into spreadsheet files > > or PDF reports. But this is based on the multi-test output from runltp. > > > > Does kirk provide the same output formats and output options as runltp? > No, but it provides something better than that: a JSON file that can be > easily parsed by most of the libraries out there. > kirk can be literally used as it is runltp, with difference of python > dependency. If you have python, you can run kirk on > host as well as on target. All features like Qemu, LTX, SSH (etc.) > supports are meant to be used by host only. The python dependency is unfortunate. In Fuego, we have carefully avoided any dependencies on the target other than those that can be provided by busybox (or toybox or toolbox). This includes limiting all test programs to either native binaries or shell scripts using just POSIX shell features. It sounds like kirk won't be useful to my project. -- Tim
> -----Original Message----- > From: automated-testing@lists.yoctoproject.org <automated-testing@lists.yoctoproject.org> On Behalf Of Cyril Hrubis > Hi! > > I don't think it's that simple. > > > > Currently, Fuego users can choose to either: > > 1. run a suite of tests (specified in the runtest file) using runltp executing on the target > > And this is something that I think is a suboptimal solution. If the run > crashes in the middle you can't easily continue with the rest of the > tests after the system under test has been rebooted. So the desing > decision for kirk was that the complete state of the testsuite execution > is being kept on the machine that drives the testing. I agree with this. That problem is actually why I developed the second solution, and another Fuego developer added skiplist support to runltp, several years ago. > I did a talk at SUSE Labs conference when I tried to explain when we are > heading and it was recorded, unfortunately it wasn't uploaded to youtube > yet, I will post the link here once it's there. > > The bottom line question is how much code from us can be reused in your > environment and this is something we have to figure out. > > > 2. run an individual test, not using runltp. > > > > In the first case, since some of the suites have a large number of tests, > > there are options in Fuego to convert the results into spreadsheet files > > or PDF reports. But this is based on the multi-test output from runltp. > > > > Does kirk provide the same output formats and output options as runltp? > > We decided to go for JSON format for generic the test results for kirk. > > However given that we store the results in python data structures > internally, it's easy to write an exporter for a format you want and > produce that directly. We can add CSV exporter for spreadsheets in 100 > lines of python. Indeed, JSON should be easier to handle. It would still require refactoring as we currently split the runltp output into different files. It would likely end up simplifying things, though. > > The JSON exported looks like this: > > https://github.com/linux-test-project/kirk/blob/master/libkirk/export.py > > > If runltp is eventually removed, I'll have to come up with a solution > > for executing suites of tests on the target, and making sure the output > > is the same as runltp (or modifying the report generation code to handle > > a new output). > > The second one is preferable. Understood. > > > Fuego supports multiple "transport" layers. ssh, serial console, > > and adb transfers are supported, as well as a few weird transports > > (such as ssh to a controller board that then transfers over serial). > > I wouldn't request kirk to support these oddball transport mechanisms, > > but if it had support for mapping it's communication mechanisms > > onto these, or supporting some pluggable mechanism for transferring > > files and executing programs (collecting stdout, stderr and return code), > > that might be useful. > > That is actually the end goal. The ltx is designed to marshal messages > over a byte oriented connection, the supported commands at the moment > can execute binaries, transfer files, etc. We intend to use it over > virtio and physical serial ports as well. So the idea is that once ltx > is started on the system uder test kirk connect to that instance to > drive the testing. It's not production ready yet, and that is something > that I have on my TODO. In that case, would kirk or python be required on the target? I presume that ltx runs on the target, and that is it a native app? What are its dependencies? > > > > > > We do have the ltx binary, which is the small dispatcher supposed to run > > > > > on the VM. And in an ideal world we would have a python library that > > > > > talks to it on the other end, as a part of kirk, that could be reused > > > > > separately. And the same for building lists of test to execute, ideally > > > > > we would have a python library that would export a simple interface so > > > > > that everyone could integrate the blocks that they really need into > > > > > their solution. > > > > > > > > Automated testing is a hard problem to solve generically and even if > > > > you do manage that, this all looks like a lot of work even just to > > > > reproduce what works today :/. > > > > > > Indeed. However I stil think that there are reusable parts that may be > > > worth putting together. > > > > Possibly. I tried for a few years to integrate Fuego and KernelCI, > > but their architectures were too different, and I eventually gave up. > > Now that KernelCI is changing, I've thought about going back > > and seeing if I could transfer anything between the projects. > > > > Usually, you have to write some kind of mapping layer, and the mappings > > turn out to be harder than expected, due to assumptions baked in to > > the architecture. > > That's what I'm hoping to avoid to keep things in a simple modules, but > I guess that we will run into a lot of implementation details that will > make things hard or even impossible... Quite likely, I think. It *is* difficult to make modular pieces that can be used by different CI systems, but I continue to have hope. :-) Just the fact that you are discussing this on this list is forward progress from where we were 10 years ago. So thank you for keeping us informed with what's going on with LTP. -- Tim
> > -----Original Message----- > > From: ltp <ltp-bounces+tim.bird=sony.com@lists.linux.it> On Behalf Of Andrea Cervesato via ltp > > Hi! > > My 2 cents below. > > On 6/7/24 23:17, Bird, Tim wrote: > > >> -----Original Message----- > > >> From: Cyril Hrubis <chrubis@suse.cz> > > >>>> I'm afraid that's not a good solution either. The end goal for kirk is > > >>>> to have a small binary locked in RAM and with realtime priority to > > >>>> execute tests and send back logs, in case of qemu over virtio, to the > > >>>> kirk. That is to make sure that logs are collected properly even when > > >>>> kernel is out of memory and in a similar situations. > > >>>> If you run kirk on the VM, reporting is not going to be reliable. > > >>> This means you're effectively mandating how ltp be run and the only > > >>> variable would be the kernel binary. Whilst I can understand that, I'm > > >>> not sure how useful us testing with this would be. > > >> Not at all. As I replied to Tim, there is no secret sauce in runltp or > > >> kirk. In the end it's a tool to execute a test binaries. If you have a > > >> system that can execute binaries, reliably transfer logs and handle > > >> kernel crashes you can as well just execute the tests yourself. All you > > >> need from us is a tooling that will produce a list(s) of tests to > > >> execute. > > > I don't think it's that simple. > > > Currently, Fuego users can choose to either: > > > 1. run a suite of tests (specified in the runtest file) using runltp executing on the target > > > 2. run an individual test, not using runltp. > > > In the first case, since some of the suites have a large number of tests, > > > there are options in Fuego to convert the results into spreadsheet files > > > or PDF reports. But this is based on the multi-test output from runltp. > > > Does kirk provide the same output formats and output options as runltp? > > No, but it provides something better than that: a JSON file that can be > > easily parsed by most of the libraries out there. > > kirk can be literally used as it is runltp, with difference of python > > dependency. If you have python, you can run kirk on > > host as well as on target. All features like Qemu, LTX, SSH (etc.) > > supports are meant to be used by host only. > The python dependency is unfortunate. In Fuego, we have carefully > avoided any dependencies on the target other than those that can > be provided by busybox (or toybox or toolbox). This includes limiting all > test programs to either native binaries or shell scripts using just POSIX > shell features. Hi Tim, as Cyril noted, having kirk on the host you would need only LTX on the target (SUT). That would avoid python on the target. LTX is very small binary (given you compile whole LTP for target). Although running kirk on the target is possible, there is not much benefit (one can just export LTPROOT, modify PATH and run test manually). I'm not trying to persuade you to take kirk (understand you have strong reasons not to take it), just FYI some test in the future will need extra tooling (e.g. support for reboot) and that functionality you either miss or will need to implement in Fuego (also probably Cyril already noted). Other thing is that we all invest time to implement our own tooling (IMHO waste of time, there is a lot of code, not just run the tests, but dmesg, coredump, restart SUT after failure, ...) instead of reusing at least a minimal subset and slowly improve that for the benefit of all. Also, the more we work with *any* shell implementation the more we see it's from the long term perspective to avoid having even slightly complex tools in shell. Simple C code avoids many problems in shell (we know POSIX shell syntax, but there are subtle differences in each implementation which show later on). At certain point we even considered writing our minimal shell implementation to avoid shell craziness (but then we just decided to write tests mostly in C). Also some tests are buggy when written in shell (e.g. cgroup tests). Kind regards, Petr > It sounds like kirk won't be useful to my project. > -- Tim
Hi Tim, > In that case, would kirk or python be required on the target? > I presume that ltx runs on the target, and that is it a native app? Yes. > What are its dependencies? IMHO no dependencies, except C toolchain. Our openSUSE spec file [1] list nothing. Running make on sources [2] show just: cc -pedantic -W -Wall -Imsgpack \ msgpack/message.c msgpack/unpack.c ltx.c main.c -o ltx I.e. just plain source code without dependencies. Kind regards, Petr [1] https://build.opensuse.org/projects/benchmark:ltp:devel/packages/ltx/files/ltx.spec?expand=1 [2] https://github.com/linux-test-project/ltx/
Hi all, looking on the discussion, we should have a wiki page which would explain the benefits we write here. And we should put this link under the deprecation message. Kind regards, Petr
> On Fri, Jun 7, 2024 at 7:50 AM Richard Purdie via lists.yoctoproject.org > <richard.purdie=linuxfoundation.org@lists.yoctoproject.org> wrote: > > On Fri, 2024-06-07 at 16:24 +0200, Petr Vorel via > > lists.yoctoproject.org wrote: > > > kirk is not perfect but already much better than old runltp script. > > > Let's deprecate runltp and propagate kirk. > > > Signed-off-by: Petr Vorel <pvorel@suse.cz> > > > --- > > > runltp | 13 +++++++++++++ > > > 1 file changed, 13 insertions(+) > > I'd note that Yocto Project's CI is still using runltp and we have no > > recipe for kirk, or any experience of using it. This does therefore > > worry me a little bit, there appears to be a lot of complexity in kirk > > we don't need. > I have begun a recipe for kirk. I know very little about the rest of how we > (Yocto Project) run ltp or how kirk is meant to be used. > https://git.yoctoproject.org/poky-contrib/log/?h=timo/kirk +1. Maybe Richard or other people who did LTP update in Yocto in the past? BTW we've just released kirk 1.3 (1.2 + few, but quite important ssh related functional fixes). Please ideally use 998df1a5aa5026c5c9b91b0caa3b1188146aa678 (hash for 1.3). Kind regards, Petr > > Cheers, > > Richard
Hi! > > No, but it provides something better than that: a JSON file that can be > > easily parsed by most of the libraries out there. > > kirk can be literally used as it is runltp, with difference of python > > dependency. If you have python, you can run kirk on > > host as well as on target. All features like Qemu, LTX, SSH (etc.) > > supports are meant to be used by host only. > > The python dependency is unfortunate. In Fuego, we have carefully > avoided any dependencies on the target other than those that can > be provided by busybox (or toybox or toolbox). This includes limiting all > test programs to either native binaries or shell scripts using just POSIX > shell features. > > It sounds like kirk won't be useful to my project. I guess that it has been cleared up in the rest of thread but let me repeat the point here. The python code is not supposed to be run on the device under test, you can do that but it's suboptimal. The ltx is the minimalistic tool that should run on the device under test and that needs only libc and some kind of byte oriented connection to the kirk runnig elsewhere. I would say that the desing is sound, the implementation needs to be finished to make it production ready. However we will not remove runltp until kirk is 100% functional and tested in our production.
Hi! > At certain point we even considered writing our minimal shell > implementation to avoid shell craziness (but then we just decided to > write tests mostly in C). I wouldn't say writing our shell, but rather bundling dash or some other lightweight shell with LTP and declaring that this is the shell to run our tests with....
Hi! > I did a talk at SUSE Labs conference when I tried to explain when we are > heading and it was recorded, unfortunately it wasn't uploaded to youtube > yet, I will post the link here once it's there. And it's finally uploaded: https://www.youtube.com/watch?v=T1ImuNr9Oxo
Hi Cyril, Andrea, > Hi! > > I did a talk at SUSE Labs conference when I tried to explain when we are > > heading and it was recorded, unfortunately it wasn't uploaded to youtube > > yet, I will post the link here once it's there. > And it's finally uploaded: > https://www.youtube.com/watch?v=T1ImuNr9Oxo Great. As I wrote It'd be great to summarize kirk + ltx architecture in LTP docs. For the start it could be just a very short article + link to this video. FYI there are some Debian users who would appreciate have kirk published via pypi, see Anders's effort: https://github.com/linux-test-project/kirk/pull/20 (Probably better to move related discussion to the ticket.) Kind regards, Petr
Hi!
Back to the original patch. The runltp script has been in a maintenance
mode (i.e. no new features) for years now, so I think that it's
reasonable to add a deprecation warning.
Reviewed-by: Cyril Hrubis <chrubis@suse.cz>
Hi Cyril, all, > Hi! > Back to the original patch. The runltp script has been in a maintenance > mode (i.e. no new features) for years now, so I think that it's > reasonable to add a deprecation warning. Thanks a lot for a review, patchset merged. Kind regards, Petr
diff --git a/runltp b/runltp index 62ab66871..4b0bd08f1 100755 --- a/runltp +++ b/runltp @@ -68,9 +68,19 @@ ################################################################################# +deprecated() +{ + echo "-------------------------------------------" + echo "INFO: runltp script is deprecated, try kirk" + echo "https://github.com/linux-test-project/kirk" + echo "-------------------------------------------" +} + setup() { + deprecated + cd `dirname $0` || \ { echo "FATAL: unable to change directory to $(dirname $0)" @@ -921,6 +931,9 @@ EOF EOF } + + deprecated + exit $VALUE }
kirk is not perfect but already much better than old runltp script. Let's deprecate runltp and propagate kirk. Signed-off-by: Petr Vorel <pvorel@suse.cz> --- runltp | 13 +++++++++++++ 1 file changed, 13 insertions(+)