| Message ID | 20231017085133.1203402-3-thaller@redhat.com |
|---|---|
| State | Rejected, archived |
| Headers | show |
| Series | add "eval-exit-code" and skip tests based on kernel version | expand |
On Tue, Oct 17, 2023 at 10:49:07AM +0200, Thomas Haller wrote: > The "table_onoff" test can only pass with certain (recent) kernels. > Conditionally exit with status 77, if "eval-exit-code" determines that > we don't have a suitable kernel version. > > In this case, we can find the fixes in: > > v6.6 : https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=c9bd26513b3a11b3adb3c2ed8a31a01a87173ff1 > v6.5.6 : https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=5e5754e9e77ce400d70ff3c30fea466c8dfe9a9f > v6.1.56 : https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=c4b0facd5c20ceae3d07018a3417f06302fa9cd1 > v5.15.135 : https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=0dcc9b4097d860d9af52db5366a8755c13468d13 I am not sure it worth this level of tracking. Soon these patches will be in upstream stable and this extra shell code will be simply deadcode in little time. > Fixes: bcca2d67656f ('tests: add test for dormant on/off/on bug') > Signed-off-by: Thomas Haller <thaller@redhat.com> > --- > tests/shell/testcases/transactions/table_onoff | 4 +++- > 1 file changed, 3 insertions(+), 1 deletion(-) > > diff --git a/tests/shell/testcases/transactions/table_onoff b/tests/shell/testcases/transactions/table_onoff > index 831d4614c1f2..0e70ad2cc3f4 100755 > --- a/tests/shell/testcases/transactions/table_onoff > +++ b/tests/shell/testcases/transactions/table_onoff > @@ -11,7 +11,9 @@ delete table ip t > EOF > > if [ $? -eq 0 ]; then > - exit 1 > + echo "Command to re-awaken a dormant table did not fail. Lacking https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=c9bd26513b3a11b3adb3c2ed8a31a01a87173ff1 ?" > + "$NFT_TEST_BASEDIR/helpers/eval-exit-code" kernel 6.6 6.5.6 6.1.56 5.15.135 > + exit $? > fi > > set -e > -- > 2.41.0 >
On Tue, 2023-10-17 at 11:04 +0200, Pablo Neira Ayuso wrote: > On Tue, Oct 17, 2023 at 10:49:07AM +0200, Thomas Haller wrote: > > The "table_onoff" test can only pass with certain (recent) kernels. > > Conditionally exit with status 77, if "eval-exit-code" determines > > that > > we don't have a suitable kernel version. > > > > In this case, we can find the fixes in: > > > > v6.6 : > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=c9bd26513b3a11b3adb3c2ed8a31a01a87173ff1 > > v6.5.6 : > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=5e5754e9e77ce400d70ff3c30fea466c8dfe9a9f > > v6.1.56 : > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=c4b0facd5c20ceae3d07018a3417f06302fa9cd1 > > v5.15.135 : > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=0dcc9b4097d860d9af52db5366a8755c13468d13 > > I am not sure it worth this level of tracking. > > Soon these patches will be in upstream stable and this extra shell > code will be simply deadcode in little time. I am not concerned about dead code in old tests that keep passing. The code was useful once, now the test passes. No need to revisit them, unless you see a real problem with them. If it would be only little time, the tests should wait. But how much is the right time? You are not waiting for your use-case, you are holding back to not to break the unknown use cases of others. IMO merging tests is good. The problem just needs a good solution. Thomas
On Tue, Oct 17, 2023 at 01:21:54PM +0200, Thomas Haller wrote: > On Tue, 2023-10-17 at 11:04 +0200, Pablo Neira Ayuso wrote: > > On Tue, Oct 17, 2023 at 10:49:07AM +0200, Thomas Haller wrote: > > > The "table_onoff" test can only pass with certain (recent) kernels. > > > Conditionally exit with status 77, if "eval-exit-code" determines > > > that > > > we don't have a suitable kernel version. > > > > > > In this case, we can find the fixes in: > > > > > > v6.6 : > > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=c9bd26513b3a11b3adb3c2ed8a31a01a87173ff1 > > > v6.5.6 : > > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=5e5754e9e77ce400d70ff3c30fea466c8dfe9a9f > > > v6.1.56 : > > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=c4b0facd5c20ceae3d07018a3417f06302fa9cd1 > > > v5.15.135 : > > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=0dcc9b4097d860d9af52db5366a8755c13468d13 > > > > I am not sure it worth this level of tracking. > > > > Soon these patches will be in upstream stable and this extra shell > > code will be simply deadcode in little time. > > I am not concerned about dead code in old tests that keep passing. > The code was useful once, now the test passes. No need to revisit them, > unless you see a real problem with them. > > If it would be only little time, the tests should wait. But how much is > the right time? You are not waiting for your use-case, you are holding > back to not to break the unknown use cases of others. > > IMO merging tests is good. The problem just needs a good solution. Apologies, I don't think this kind of hints is worth. It might be a bug that resurrects in the future and then the suggestion that a commit is missing will be misleading.
On Tue, 2023-10-17 at 13:40 +0200, Pablo Neira Ayuso wrote: > On Tue, Oct 17, 2023 at 01:21:54PM +0200, Thomas Haller wrote: > > On Tue, 2023-10-17 at 11:04 +0200, Pablo Neira Ayuso wrote: > > > On Tue, Oct 17, 2023 at 10:49:07AM +0200, Thomas Haller wrote: > > > > The "table_onoff" test can only pass with certain (recent) > > > > kernels. > > > > Conditionally exit with status 77, if "eval-exit-code" > > > > determines > > > > that > > > > we don't have a suitable kernel version. > > > > > > > > In this case, we can find the fixes in: > > > > > > > > v6.6 : > > > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=c9bd26513b3a11b3adb3c2ed8a31a01a87173ff1 > > > > v6.5.6 : > > > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=5e5754e9e77ce400d70ff3c30fea466c8dfe9a9f > > > > v6.1.56 : > > > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=c4b0facd5c20ceae3d07018a3417f06302fa9cd1 > > > > v5.15.135 : > > > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=0dcc9b4097d860d9af52db5366a8755c13468d13 > > > > > > I am not sure it worth this level of tracking. > > > > > > Soon these patches will be in upstream stable and this extra > > > shell > > > code will be simply deadcode in little time. > > > > I am not concerned about dead code in old tests that keep passing. > > The code was useful once, now the test passes. No need to revisit > > them, > > unless you see a real problem with them. > > > > If it would be only little time, the tests should wait. But how > > much is > > the right time? You are not waiting for your use-case, you are > > holding > > back to not to break the unknown use cases of others. > > > > IMO merging tests is good. The problem just needs a good solution. > > Apologies, I don't think this kind of hints is worth. Which hint do you mean? Do you mean the commit message, or the echo "Command to re-awaken a dormant table did not fail. Lacking https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=c9bd26513b3a11b3adb3c2ed8a31a01a87173ff1 ?" The echo is really just an elaborate code-comment. But it also ends up in "testout.log", which makes it better. I don't mind rewording commit message or the echo statement (or even dropping it entirely). But I find this information more useful than not. Thomas
On Tue, Oct 17, 2023 at 02:43:56PM +0200, Thomas Haller wrote: > On Tue, 2023-10-17 at 13:40 +0200, Pablo Neira Ayuso wrote: > > On Tue, Oct 17, 2023 at 01:21:54PM +0200, Thomas Haller wrote: > > > On Tue, 2023-10-17 at 11:04 +0200, Pablo Neira Ayuso wrote: > > > > On Tue, Oct 17, 2023 at 10:49:07AM +0200, Thomas Haller wrote: > > > > > The "table_onoff" test can only pass with certain (recent) > > > > > kernels. > > > > > Conditionally exit with status 77, if "eval-exit-code" > > > > > determines > > > > > that > > > > > we don't have a suitable kernel version. > > > > > > > > > > In this case, we can find the fixes in: > > > > > > > > > > v6.6 : > > > > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=c9bd26513b3a11b3adb3c2ed8a31a01a87173ff1 > > > > > v6.5.6 : > > > > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=5e5754e9e77ce400d70ff3c30fea466c8dfe9a9f > > > > > v6.1.56 : > > > > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=c4b0facd5c20ceae3d07018a3417f06302fa9cd1 > > > > > v5.15.135 : > > > > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=0dcc9b4097d860d9af52db5366a8755c13468d13 > > > > > > > > I am not sure it worth this level of tracking. > > > > > > > > Soon these patches will be in upstream stable and this extra > > > > shell > > > > code will be simply deadcode in little time. > > > > > > I am not concerned about dead code in old tests that keep passing. > > > The code was useful once, now the test passes. No need to revisit > > > them, > > > unless you see a real problem with them. > > > > > > If it would be only little time, the tests should wait. But how > > > much is > > > the right time? You are not waiting for your use-case, you are > > > holding > > > back to not to break the unknown use cases of others. > > > > > > IMO merging tests is good. The problem just needs a good solution. > > > > Apologies, I don't think this kind of hints is worth. > > > Which hint do you mean? > > Do you mean the commit message, or the > > echo "Command to re-awaken a dormant table did not fail. Lacking https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=c9bd26513b3a11b3adb3c2ed8a31a01a87173ff1 ?" > > The echo is really just an elaborate code-comment. But it also ends up > in "testout.log", which makes it better. > > > I don't mind rewording commit message or the echo statement (or even > dropping it entirely). But I find this information more useful than > not. But I still don't get the point on skipping this test in older kernels, this test tells you your kernel is buggy. Who is going to run this in older kernels? We only run these tests these days. Skipping a test that tells you that you kernel is buggy is misleading?
diff --git a/tests/shell/testcases/transactions/table_onoff b/tests/shell/testcases/transactions/table_onoff index 831d4614c1f2..0e70ad2cc3f4 100755 --- a/tests/shell/testcases/transactions/table_onoff +++ b/tests/shell/testcases/transactions/table_onoff @@ -11,7 +11,9 @@ delete table ip t EOF if [ $? -eq 0 ]; then - exit 1 + echo "Command to re-awaken a dormant table did not fail. Lacking https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=c9bd26513b3a11b3adb3c2ed8a31a01a87173ff1 ?" + "$NFT_TEST_BASEDIR/helpers/eval-exit-code" kernel 6.6 6.5.6 6.1.56 5.15.135 + exit $? fi set -e
The "table_onoff" test can only pass with certain (recent) kernels. Conditionally exit with status 77, if "eval-exit-code" determines that we don't have a suitable kernel version. In this case, we can find the fixes in: v6.6 : https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=c9bd26513b3a11b3adb3c2ed8a31a01a87173ff1 v6.5.6 : https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=5e5754e9e77ce400d70ff3c30fea466c8dfe9a9f v6.1.56 : https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=c4b0facd5c20ceae3d07018a3417f06302fa9cd1 v5.15.135 : https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=0dcc9b4097d860d9af52db5366a8755c13468d13 Fixes: bcca2d67656f ('tests: add test for dormant on/off/on bug') Signed-off-by: Thomas Haller <thaller@redhat.com> --- tests/shell/testcases/transactions/table_onoff | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)