Message ID | 20221201191314.1916813-1-numans@ovn.org |
---|---|
State | Changes Requested |
Headers | show |
Series | CI: Add ovn-fake-multinode jobs | expand |
Context | Check | Description |
---|---|---|
ovsrobot/apply-robot | fail | apply and check: fail |
Bleep bloop. Greetings Numan Siddique, I am a robot and I have tried out your patch. Thanks for your contribution. I encountered some error that I wasn't expecting. See the details below. git-am: error: sha1 information is lacking or useless (.github/workflows/ovn-fake-multinode-tests.yml). error: could not build fake ancestor hint: Use 'git am --show-current-patch=diff' to see the failed patch Patch failed at 0001 Add fake multinode system tests. When you have resolved this problem, run "git am --continue". If you prefer to skip this patch, run "git am --skip" instead. To restore the original branch and stop patching, run "git am --abort". Please check this out. If you feel there has been an error, please email aconole@redhat.com Thanks, 0-day Robot
Hi Numan, On Thu, Dec 1, 2022 at 8:13 PM <numans@ovn.org> wrote: > From: Numan Siddique <numans@ovn.org> > > This patch adds a simple system test using ovn-fake-multinode > setup. The tests can be run as - 'make check-multinode' > > Before running these tests, user should deploy fake_multinode setup > by running 'ovn_cluster.sh start'. > > This test suite is also triggered for the newly added fake multinode CI > job. > > The fake multinode system tests suite can be enhanced further for new > features and to cover multi node scenarios. > > Signed-off-by: Numan Siddique <numans@ovn.org> > --- > .../workflows/ovn-fake-multinode-tests.yml | 57 ++++++ > tests/automake.mk | 28 ++- > tests/multinode-macros.at | 189 ++++++++++++++++++ > tests/multinode-testsuite.at | 27 +++ > tests/multinode.at | 74 +++++++ > 5 files changed, 374 insertions(+), 1 deletion(-) > create mode 100644 tests/multinode-macros.at > create mode 100644 tests/multinode-testsuite.at > create mode 100644 tests/multinode.at > > diff --git a/.github/workflows/ovn-fake-multinode-tests.yml > b/.github/workflows/ovn-fake-multinode-tests.yml > index 3727b9835..c24eb2965 100644 > --- a/.github/workflows/ovn-fake-multinode-tests.yml > +++ b/.github/workflows/ovn-fake-multinode-tests.yml > @@ -86,6 +86,12 @@ jobs: > OS_IMAGE: "fedora:36" > CENTRAL_IMAGE: ${{ matrix.cfg.central_image }} > ENABLE_SSL: no > + CC: gcc > + OPTS: "--disable-ssl" > + dependencies: | > + automake libtool gcc bc libjemalloc2 libjemalloc-dev \ > + libssl-dev llvm-dev libelf-dev libnuma-dev libpcap-dev \ > + selinux-policy-dev ncat python3-scapy isc-dhcp-server > # https://github.com/actions/runner-images/issues/6282 > XDG_RUNTIME_DIR: '' > > @@ -145,6 +151,57 @@ jobs: > cd ovn-fake-multinode > sudo ./.ci/test_basic.sh > > + - name: install required dependencies > + run: sudo apt install -y ${{ env.dependencies }} > nit: It would be better to install dependencies before other steps. > + > + - name: install libunbound libunwind > + run: sudo apt install -y libunbound-dev libunwind-dev > + > + - name: update PATH > + run: | > + echo "$HOME/bin" >> $GITHUB_PATH > + echo "$HOME/.local/bin" >> $GITHUB_PATH > + > + - name: set up python > + uses: actions/setup-python@v4 > + with: > + python-version: '3.x' > + > + - name: Check out ovn > + uses: actions/checkout@v3 > + with: > + path: 'ovn' > + submodules: recursive > + > + - name: Build OVN and trigger fake-multinode system tests > + run: | > + set -x > + cd ovn > + ./.ci/linux-prepare.sh > + ./.ci/linux-build.sh > + sudo make check-multinode > + > + - name: copy logs on failure > + if: failure() || cancelled() > + run: | > + # upload-artifact@v3 throws exceptions if it tries to upload > socket > + # files and we could have some socket files in testsuite.dir. > + # Also, upload-artifact@v3 doesn't work well enough with > wildcards. > + # So, we're just archiving everything here to avoid any issues. > + mkdir logs > + cp ovn/config.log ./logs/ > + # multinode tests are run as root, need to adjust permissions. > + sudo chmod -R +r ovn/tests/multinode-testsuite.dir.* || true > + cp -r ovn/tests/multinode-testsuite.dir.* ./logs/ || true > + tar -czvf logs.tgz logs/ > + > + - name: upload logs on failure > + if: failure() || cancelled() > + uses: actions/upload-artifact@v3 > + with: > + name: logs-linux-${{ join(matrix.cfg.*, '-') }} > + path: logs.tgz > + > - name: Stop cluster > run: | > cd ovn-fake-multinode > diff --git a/tests/automake.mk b/tests/automake.mk > index dce9c9108..88cda5f20 100644 > --- a/tests/automake.mk > +++ b/tests/automake.mk > @@ -5,10 +5,12 @@ EXTRA_DIST += \ > $(SYSTEM_KMOD_TESTSUITE_AT) \ > $(SYSTEM_USERSPACE_TESTSUITE_AT) \ > $(PERF_TESTSUITE_AT) \ > + $(MULTINODE_TESTSUITE_AT) \ > $(TESTSUITE) \ > $(SYSTEM_KMOD_TESTSUITE) \ > $(SYSTEM_USERSPACE_TESTSUITE) \ > $(PERF_TESTSUITE) \ > + $(MULTINODE_TESTSUITE) \ > tests/atlocal.in \ > $(srcdir)/package.m4 \ > $(srcdir)/tests/testsuite \ > @@ -61,6 +63,11 @@ PERF_TESTSUITE_AT = \ > tests/perf-testsuite.at \ > tests/perf-northd.at > > +MULTINODE_TESTSUITE_AT = \ > + tests/multinode-testsuite.at \ > + tests/multinode-macros.at \ > + tests/multinode.at > + > check_SCRIPTS += tests/atlocal > > TESTSUITE = $(srcdir)/tests/testsuite > @@ -72,7 +79,9 @@ PERF_TESTSUITE = $(srcdir)/tests/perf-testsuite > PERF_TESTSUITE_DIR = $(abs_top_builddir)/tests/perf-testsuite.dir > PERF_TESTSUITE_RESULTS = $(PERF_TESTSUITE_DIR)/results > DISTCLEANFILES += tests/atconfig tests/atlocal > - > +MULTINODE_TESTSUITE = $(srcdir)/tests/multinode-testsuite > +MULTINODE_TESTSUITE_DIR = > $(abs_top_builddir)/tests/multinode-testsuite.dir > +MULTINODE_TESTSUITE_RESULTS = $(MULTINODE_TESTSUITE_DIR)/results > AUTOTEST_PATH = > $(ovs_builddir)/utilities:$(ovs_builddir)/vswitchd:$(ovs_builddir)/ovsdb:$(ovs_builddir)/vtep:tests:$(PTHREAD_WIN32_DIR_DLL):$(SSL_DIR):controller-vtep:northd:utilities:controller:ic > > export ovs_srcdir > @@ -82,6 +91,7 @@ check-local: > "$$@" $(TESTSUITEFLAGS) || \ > (test -z "$$(find $(TESTSUITE_DIR) -name 'sanitizers.*')" && \ > test X'$(RECHECK)' = Xyes && "$$@" --recheck) > + > nit: Extra empty line? > > # Python Coverage support. > # Requires coverage.py http://nedbatchelder.com/code/coverage/. > @@ -197,6 +207,18 @@ check-perf: all > @echo > @echo "Results can be found in $(PERF_TESTSUITE_RESULTS)" > > +check-multinode: all > + @mkdir -p $(MULTINODE_TESTSUITE_DIR) > + @echo > $(MULTINODE_TESTSUITE_RESULTS) > + set $(SHELL) '$(MULTINODE_TESTSUITE)' -C tests > AUTOTEST_PATH='$(AUTOTEST_PATH)'; \ > + "$$@" $(TESTSUITEFLAGS) -j1 || (test X'$(RECHECK)' = Xyes && "$$@" > --recheck) > + @echo > + @echo '## -------------------- ##' > + @echo '## Multinode test Results ##' > + @echo '## -------------------- ##' > + @cat $(MULTINODE_TESTSUITE_RESULTS) > + @echo > + @echo "Results can be found in $(MULTINODE_TESTSUITE_RESULTS)" > > AUTOTEST = $(AUTOM4TE) --language=autotest > > @@ -223,6 +245,10 @@ $(PERF_TESTSUITE): package.m4 $(PERF_TESTSUITE_AT) > $(COMMON_MACROS_AT) > $(AM_V_GEN)$(AUTOTEST) -I '$(srcdir)' -o $@.tmp $@.at > $(AM_V_at)mv $@.tmp $@ > > +$(MULTINODE_TESTSUITE): package.m4 $(MULTINODE_TESTSUITE_AT) > $(COMMON_MACROS_AT) > + $(AM_V_GEN)$(AUTOTEST) -I '$(srcdir)' -o $@.tmp $@.at > + $(AM_V_at)mv $@.tmp $@ > + > # The `:;' works around a Bash 3.2 bug when the output is not writeable. > $(srcdir)/package.m4: $(top_srcdir)/configure.ac > $(AM_V_GEN):;{ \ > diff --git a/tests/multinode-macros.at b/tests/multinode-macros.at > new file mode 100644 > index 000000000..88a3816eb > --- /dev/null > +++ b/tests/multinode-macros.at > @@ -0,0 +1,189 @@ > +# > +# > + > +# M_NS_EXEC([fake_node], [namespace], [command]) > +# > +# Execute 'command' in 'namespace' > +m4_define([M_NS_EXEC], > + [podman exec $1 ip netns exec $2 $3]) > Shouldn't we use -i every time we call "podman exec"? > + > +# M_NS_CHECK_EXEC(fake_node], [namespace], [command], other_params...) > nit: Missing [ before "fake_node]". > +# > +# Wrapper for AT_CHECK that executes 'command' inside 'fake_node''s > namespace'. > +# 'other_params' as passed as they are to AT_CHECK. > +m4_define([M_NS_CHECK_EXEC], > + [ AT_CHECK([M_NS_EXEC([$1], [$2], [$3])], > m4_shift(m4_shift(m4_shift($@)))) ] > +) > + > +OVS_START_SHELL_HELPERS > + > +m_as() { > + c=$1 > + shift > + podman exec $c "$@" > +} > + > +m_central_as () { > + podman exec ovn-central "$@" > +} > + > +check_fake_multinode_setup() { > + check m_as ovn-central ovn-nbctl --wait=hv sync > + AT_CHECK([m_as ovn-chassis-1 ovn-appctl -t ovn-controller version], > [0], [ignore]) > + AT_CHECK([m_as ovn-chassis-2 ovn-appctl -t ovn-controller version], > [0], [ignore]) > + AT_CHECK([m_as ovn-gw-1 ovn-appctl -t ovn-controller version], [0], > [ignore]) > +} > + > +cleanup_multinode_resources() { > + m_as ovn-central rm -f /etc/ovn/ovnnb_db.db > + m_as ovn-central /usr/share/ovn/scripts/ovn-ctl restart_northd > + check m_as ovn-central ovn-nbctl --wait=hv sync > + for c in ovn-chassis-1 ovn-chassis-2 ovn-gw-1 > + do > + m_as $c ovs-vsctl del-br br-int > + m_as $c ip --all netns delete > + done > +} > + > +multinode_nbctl () { > + m_as ovn-central ovn-nbctl "$@" > +} > + > +# count_rows TABLE [CONDITION...] > +# > +# Prints the number of rows in TABLE (that satisfy CONDITION). > +# Uses the southbound db by default; set DB=nb for the northbound > database. > +m_count_rows() { > + local db=$(parse_db $1) table=$(parse_table $1); shift > + m_central_as ovn-${db}ctl --format=table --no-headings find $table > "$@" | wc -l > +} > + > +# check_row_count [DATABASE:]TABLE COUNT [CONDITION...] > +# > +# Checks that TABLE contains COUNT rows (that satisfy CONDITION). > +# The default DATABASE is "sb". > +m_check_row_count() { > + local db=$(parse_db $1) table=$(parse_table $1); shift > + local count=$1; shift > + local found=$(m_count_rows $c $db:$table "$@") > + echo > + echo "Checking for $count rows in $db $table${1+ with $*}... found > $found" > + if test "$count" != "$found"; then > + m_central_as ovn-${db}ctl list $table > + AT_FAIL_IF([:]) > + fi > +} > + > +# wait_row_count [DATABASE:]TABLE COUNT [CONDITION...] > +# > +# Waits until TABLE contains COUNT rows (that satisfy CONDITION). > +# The default DATABASE is "sb". > +m_wait_row_count() { > + local db=$(parse_db $1) table=$(parse_table $1); shift > + local count=$1; shift > + local a=$1 b=$2 c=$3 d=$4 e=$5 > + echo "Waiting until $count rows in $db $table${1+ with $*}..." > + OVS_WAIT_UNTIL([test $count = $(m_count_rows $db:$table $a $b $c $d > $e)],[ > + echo "$db table $table has the following rows. $(m_count_rows > $db:$table $a $b $c $d $e) rows match instead of expected $count:" > + m_central_as ovn-${db}ctl list $table]) > +} > + > +# multinode_wait_column EXPECTED [DATABASE:]TABLE [COLUMN [CONDITION...]] > +# > +# Wait until all of the values of COLUMN in the rows of TABLE (that > +# satisfy CONDITION) equal EXPECTED (ignoring order). > +# > +# The default DATABASE is "sb". > +# > +# COLUMN defaults to _uuid if unspecified. > +m_wait_column() { > + local expected=$(for d in $1; do echo $d; done | sort) > + local db=$(parse_db $2) table=$(parse_table $2) column=${3-_uuid}; > shift; shift; shift > + local a=$1 b=$2 c=$3 d=$4 e=$5 > + > + echo > + echo "Waiting until $column in $db $table${1+ with $*} is > $expected..." > + OVS_WAIT_UNTIL([ > + found=$(m_central_as ovn-${db}ctl --bare --columns $column find > $table $a $b $c $d $e) > + found=$(for d in $found; do echo $d; done | sort) > + test "$expected" = "$found" > + ], [ > + echo "$column in $db table $table has value $found, from the > following rows:" > + m_central_as ovn-${db}ctl list $table]) > +} > + > +# fetch_column [DATABASE:]TABLE COLUMN [CONDITION...] > +# > +# Fetches and prints all the values of COLUMN in the rows of TABLE > +# (that satisfy CONDITION), sorting the results lexicographically. > +# The default DATABASE is "sb". > +m_fetch_column() { > + local db=$(parse_db $1) table=$(parse_table $1) column=${2-_uuid}; > shift; shift > + # Using "echo" removes spaces and newlines. > + echo $(m_central_as ovn-${db}ctl --bare --columns $column find $table > "$@" | sort) > +} > + > +# check_column EXPECTED [DATABASE:]TABLE COLUMN [CONDITION...] > +# > +# Fetches all of the values of COLUMN in the rows of TABLE (that > +# satisfy CONDITION), and compares them against EXPECTED (ignoring > +# order). > +# > +# The default DATABASE is "sb". > +m_check_column() { > + local expected=$1 db=$(parse_db $2) table=$(parse_table $2) > column=${3-_uuid}; shift; shift; shift > + local found=$(m_central_as ovn-${db}ctl --bare --columns $column find > $table "$@") > + > + # Sort the expected and found values. > + local found=$(for d in $found; do echo $d; done | sort) > + local expected=$(for d in $expected; do echo $d; done | sort) > + > + echo > + echo "Checking values in $db $table${1+ with $*} against $expected... > found $found" > + if test "$found" != "$expected"; then > + m_central_as ovn-${db}ctl list $table > + AT_FAIL_IF([:]) > + fi > +} > + > +# wait_column EXPECTED [DATABASE:]TABLE [COLUMN [CONDITION...]] > +# > +# Wait until all of the values of COLUMN in the rows of TABLE (that > +# satisfy CONDITION) equal EXPECTED (ignoring order). > +# > +# The default DATABASE is "sb". > +# > +# COLUMN defaults to _uuid if unspecified. > +m_wait_column() { > + local expected=$(for d in $1; do echo $d; done | sort) > + local db=$(parse_db $2) table=$(parse_table $2) column=${3-_uuid}; > shift; shift; shift > + local a=$1 b=$2 c=$3 d=$4 e=$5 > + > + echo > + echo "Waiting until $column in $db $table${1+ with $*} is > $expected..." > + OVS_WAIT_UNTIL([ > + found=$(m_central_as ovn-${db}ctl --bare --columns $column find > $table $a $b $c $d $e) > + found=$(for d in $found; do echo $d; done | sort) > + test "$expected" = "$found" > + ], [ > + echo "$column in $db table $table has value $found, from the > following rows:" > + m_central_as ovn-${db}ctl list $table]) > +} > + > +# wait_for_ports_up [PORT...] > +# > +# With arguments, waits for specified Logical_Switch_Ports to come up. > +# Without arguments, waits for all "plain" and router > +# Logical_Switch_Ports to come up. > +m_wait_for_ports_up() { > + if test $# = 0; then > + m_wait_row_count nb:Logical_Switch_Port 0 up!=true type='""' > + m_wait_row_count nb:Logical_Switch_Port 0 up!=true type=router > + else > + for port; do > + m_wait_row_count nb:Logical_Switch_Port 1 up=true name=$port > + done > + fi > +} > + > +OVS_END_SHELL_HELPERS > diff --git a/tests/multinode-testsuite.at b/tests/multinode-testsuite.at > new file mode 100644 > index 000000000..ea10b0276 > --- /dev/null > +++ b/tests/multinode-testsuite.at > @@ -0,0 +1,27 @@ > +AT_INIT > + > +AT_COPYRIGHT([Copyright (c) 2022 Red Hat, > + > +Licensed under the Apache License, Version 2.0 (the "License"); > +you may not use this file except in compliance with the License. > +You may obtain a copy of the License at: > + > + http://www.apache.org/licenses/LICENSE-2.0 > + > +Unless required by applicable law or agreed to in writing, software > +distributed under the License is distributed on an "AS IS" BASIS, > +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. > +See the License for the specific language governing permissions and > +limitations under the License.]) > + > +m4_ifdef([AT_COLOR_TESTS], [AT_COLOR_TESTS]) > +AT_ARG_OPTION([rebuild], [Do not use cached versions of databases]) > + > +m4_include([tests/ovs-macros.at]) > +m4_include([tests/ovsdb-macros.at]) > +m4_include([tests/ofproto-macros.at]) > +m4_include([tests/ovn-macros.at]) > +m4_include([tests/system-common-macros.at]) > +m4_include([tests/multinode-macros.at]) > + > +m4_include([tests/multinode.at]) > diff --git a/tests/multinode.at b/tests/multinode.at > new file mode 100644 > index 000000000..754e488d6 > --- /dev/null > +++ b/tests/multinode.at > @@ -0,0 +1,74 @@ > +AT_BANNER([ovn multinode system tests using ovn-fake-multinode]) > + > +AT_SETUP([ovn multinode basic test]) > + > +# Check that ovn-fake-multinode setup is up and running > +check_fake_multinode_setup > + > +# Delete the multinode NB and OVS resources before starting the test. > +cleanup_multinode_resources > + > +# Test East-West switching > +check multinode_nbctl ls-add sw0 > +check multinode_nbctl lsp-add sw0 sw0-port1 > +check multinode_nbctl lsp-set-addresses sw0-port1 "50:54:00:00:00:03 > 10.0.0.3 1000::3" > +check multinode_nbctl lsp-add sw0 sw0-port2 > +check multinode_nbctl lsp-set-addresses sw0-port2 "50:54:00:00:00:04 > 10.0.0.4 1000::4" > + > +m_as ovn-chassis-1 /data/create_fake_vm.sh sw0-port1 sw0p1 > 50:54:00:00:00:03 10.0.0.3 24 10.0.0.1 1000::3/64 1000::a > +m_as ovn-chassis-2 /data/create_fake_vm.sh sw0-port2 sw0p2 > 50:54:00:00:00:04 10.0.0.4 24 10.0.0.1 1000::4/64 1000::a > + > +m_wait_for_ports_up > + > +M_NS_CHECK_EXEC([ovn-chassis-1], [sw0p1], [ping -q -c 3 -i 0.3 -w 2 > 10.0.0.4 | FORMAT_PING], \ > +[0], [dnl > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > +]) > + > +# Add ACLs to drop all traffic > +check multinode_nbctl pg-add pg0 sw0-port1 sw0-port2 > +check multinode_nbctl acl-add pg0 to-lport 1001 "outport == @pg0 && ip4" > drop > +check multinode_nbctl --wait=hv sync > + > +M_NS_CHECK_EXEC([ovn-chassis-1], [sw0p1], [ping -q -c 3 -i 0.3 -w 2 > 10.0.0.4], \ > +[1], [ignore]) > + > +# Add ACLs to allow icmp traffic > +check multinode_nbctl acl-add pg0 to-lport 1002 "outport == @pg0 && ip4 > && icmp" allow-related > +check multinode_nbctl --wait=hv sync > + > +M_NS_CHECK_EXEC([ovn-chassis-1], [sw0p1], [ping -q -c 3 -i 0.3 -w 2 > 10.0.0.4 | FORMAT_PING], \ > +[0], [dnl > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > +]) > + > + > +# Create the second logical switch with one port > +check multinode_nbctl ls-add sw1 > +check multinode_nbctl lsp-add sw1 sw1-port1 > +check multinode_nbctl lsp-set-addresses sw1-port1 "40:54:00:00:00:03 > 20.0.0.3 2000::3" > + > +# Create a logical router and attach both logical switches > +check multinode_nbctl lr-add lr0 > +check multinode_nbctl lrp-add lr0 lr0-sw0 00:00:00:00:ff:01 10.0.0.1/24 > 1000::a/64 > +check multinode_nbctl lsp-add sw0 sw0-lr0 > +check multinode_nbctl lsp-set-type sw0-lr0 router > +check multinode_nbctl lsp-set-addresses sw0-lr0 router > +check multinode_nbctl lsp-set-options sw0-lr0 router-port=lr0-sw0 > + > +check multinode_nbctl lrp-add lr0 lr0-sw1 00:00:00:00:ff:02 20.0.0.1/24 > 2000::a/64 > +check multinode_nbctl lsp-add sw1 sw1-lr0 > +check multinode_nbctl lsp-set-type sw1-lr0 router > +check multinode_nbctl lsp-set-addresses sw1-lr0 router > +check multinode_nbctl lsp-set-options sw1-lr0 router-port=lr0-sw1 > + > +m_as ovn-chassis-2 /data/create_fake_vm.sh sw1-port1 sw1p1 > 40:54:00:00:00:03 20.0.0.3 24 20.0.0.1 2000::4/64 1000::a > + > +m_wait_for_ports_up sw1-port1 > + > +M_NS_CHECK_EXEC([ovn-chassis-1], [sw0p1], [ping -q -c 3 -i 0.3 -w 2 > 20.0.0.3 | FORMAT_PING], \ > +[0], [dnl > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > +]) > + > +AT_CLEANUP > -- > 2.38.1 > > _______________________________________________ > dev mailing list > dev@openvswitch.org > https://mail.openvswitch.org/mailman/listinfo/ovs-dev > > Other than that it looks good. Thanks, Ales
diff --git a/.github/workflows/ovn-fake-multinode-tests.yml b/.github/workflows/ovn-fake-multinode-tests.yml index 3727b9835..c24eb2965 100644 --- a/.github/workflows/ovn-fake-multinode-tests.yml +++ b/.github/workflows/ovn-fake-multinode-tests.yml @@ -86,6 +86,12 @@ jobs: OS_IMAGE: "fedora:36" CENTRAL_IMAGE: ${{ matrix.cfg.central_image }} ENABLE_SSL: no + CC: gcc + OPTS: "--disable-ssl" + dependencies: | + automake libtool gcc bc libjemalloc2 libjemalloc-dev \ + libssl-dev llvm-dev libelf-dev libnuma-dev libpcap-dev \ + selinux-policy-dev ncat python3-scapy isc-dhcp-server # https://github.com/actions/runner-images/issues/6282 XDG_RUNTIME_DIR: '' @@ -145,6 +151,57 @@ jobs: cd ovn-fake-multinode sudo ./.ci/test_basic.sh + - name: install required dependencies + run: sudo apt install -y ${{ env.dependencies }} + + - name: install libunbound libunwind + run: sudo apt install -y libunbound-dev libunwind-dev + + - name: update PATH + run: | + echo "$HOME/bin" >> $GITHUB_PATH + echo "$HOME/.local/bin" >> $GITHUB_PATH + + - name: set up python + uses: actions/setup-python@v4 + with: + python-version: '3.x' + + - name: Check out ovn + uses: actions/checkout@v3 + with: + path: 'ovn' + submodules: recursive + + - name: Build OVN and trigger fake-multinode system tests + run: | + set -x + cd ovn + ./.ci/linux-prepare.sh + ./.ci/linux-build.sh + sudo make check-multinode + + - name: copy logs on failure + if: failure() || cancelled() + run: | + # upload-artifact@v3 throws exceptions if it tries to upload socket + # files and we could have some socket files in testsuite.dir. + # Also, upload-artifact@v3 doesn't work well enough with wildcards. + # So, we're just archiving everything here to avoid any issues. + mkdir logs + cp ovn/config.log ./logs/ + # multinode tests are run as root, need to adjust permissions. + sudo chmod -R +r ovn/tests/multinode-testsuite.dir.* || true + cp -r ovn/tests/multinode-testsuite.dir.* ./logs/ || true + tar -czvf logs.tgz logs/ + + - name: upload logs on failure + if: failure() || cancelled() + uses: actions/upload-artifact@v3 + with: + name: logs-linux-${{ join(matrix.cfg.*, '-') }} + path: logs.tgz + - name: Stop cluster run: | cd ovn-fake-multinode diff --git a/tests/automake.mk b/tests/automake.mk index dce9c9108..88cda5f20 100644 --- a/tests/automake.mk +++ b/tests/automake.mk @@ -5,10 +5,12 @@ EXTRA_DIST += \ $(SYSTEM_KMOD_TESTSUITE_AT) \ $(SYSTEM_USERSPACE_TESTSUITE_AT) \ $(PERF_TESTSUITE_AT) \ + $(MULTINODE_TESTSUITE_AT) \ $(TESTSUITE) \ $(SYSTEM_KMOD_TESTSUITE) \ $(SYSTEM_USERSPACE_TESTSUITE) \ $(PERF_TESTSUITE) \ + $(MULTINODE_TESTSUITE) \ tests/atlocal.in \ $(srcdir)/package.m4 \ $(srcdir)/tests/testsuite \ @@ -61,6 +63,11 @@ PERF_TESTSUITE_AT = \ tests/perf-testsuite.at \ tests/perf-northd.at +MULTINODE_TESTSUITE_AT = \ + tests/multinode-testsuite.at \ + tests/multinode-macros.at \ + tests/multinode.at + check_SCRIPTS += tests/atlocal TESTSUITE = $(srcdir)/tests/testsuite @@ -72,7 +79,9 @@ PERF_TESTSUITE = $(srcdir)/tests/perf-testsuite PERF_TESTSUITE_DIR = $(abs_top_builddir)/tests/perf-testsuite.dir PERF_TESTSUITE_RESULTS = $(PERF_TESTSUITE_DIR)/results DISTCLEANFILES += tests/atconfig tests/atlocal - +MULTINODE_TESTSUITE = $(srcdir)/tests/multinode-testsuite +MULTINODE_TESTSUITE_DIR = $(abs_top_builddir)/tests/multinode-testsuite.dir +MULTINODE_TESTSUITE_RESULTS = $(MULTINODE_TESTSUITE_DIR)/results AUTOTEST_PATH = $(ovs_builddir)/utilities:$(ovs_builddir)/vswitchd:$(ovs_builddir)/ovsdb:$(ovs_builddir)/vtep:tests:$(PTHREAD_WIN32_DIR_DLL):$(SSL_DIR):controller-vtep:northd:utilities:controller:ic export ovs_srcdir @@ -82,6 +91,7 @@ check-local: "$$@" $(TESTSUITEFLAGS) || \ (test -z "$$(find $(TESTSUITE_DIR) -name 'sanitizers.*')" && \ test X'$(RECHECK)' = Xyes && "$$@" --recheck) + # Python Coverage support. # Requires coverage.py http://nedbatchelder.com/code/coverage/. @@ -197,6 +207,18 @@ check-perf: all @echo @echo "Results can be found in $(PERF_TESTSUITE_RESULTS)" +check-multinode: all + @mkdir -p $(MULTINODE_TESTSUITE_DIR) + @echo > $(MULTINODE_TESTSUITE_RESULTS) + set $(SHELL) '$(MULTINODE_TESTSUITE)' -C tests AUTOTEST_PATH='$(AUTOTEST_PATH)'; \ + "$$@" $(TESTSUITEFLAGS) -j1 || (test X'$(RECHECK)' = Xyes && "$$@" --recheck) + @echo + @echo '## -------------------- ##' + @echo '## Multinode test Results ##' + @echo '## -------------------- ##' + @cat $(MULTINODE_TESTSUITE_RESULTS) + @echo + @echo "Results can be found in $(MULTINODE_TESTSUITE_RESULTS)" AUTOTEST = $(AUTOM4TE) --language=autotest @@ -223,6 +245,10 @@ $(PERF_TESTSUITE): package.m4 $(PERF_TESTSUITE_AT) $(COMMON_MACROS_AT) $(AM_V_GEN)$(AUTOTEST) -I '$(srcdir)' -o $@.tmp $@.at $(AM_V_at)mv $@.tmp $@ +$(MULTINODE_TESTSUITE): package.m4 $(MULTINODE_TESTSUITE_AT) $(COMMON_MACROS_AT) + $(AM_V_GEN)$(AUTOTEST) -I '$(srcdir)' -o $@.tmp $@.at + $(AM_V_at)mv $@.tmp $@ + # The `:;' works around a Bash 3.2 bug when the output is not writeable. $(srcdir)/package.m4: $(top_srcdir)/configure.ac $(AM_V_GEN):;{ \ diff --git a/tests/multinode-macros.at b/tests/multinode-macros.at new file mode 100644 index 000000000..88a3816eb --- /dev/null +++ b/tests/multinode-macros.at @@ -0,0 +1,189 @@ +# +# + +# M_NS_EXEC([fake_node], [namespace], [command]) +# +# Execute 'command' in 'namespace' +m4_define([M_NS_EXEC], + [podman exec $1 ip netns exec $2 $3]) + +# M_NS_CHECK_EXEC(fake_node], [namespace], [command], other_params...) +# +# Wrapper for AT_CHECK that executes 'command' inside 'fake_node''s namespace'. +# 'other_params' as passed as they are to AT_CHECK. +m4_define([M_NS_CHECK_EXEC], + [ AT_CHECK([M_NS_EXEC([$1], [$2], [$3])], m4_shift(m4_shift(m4_shift($@)))) ] +) + +OVS_START_SHELL_HELPERS + +m_as() { + c=$1 + shift + podman exec $c "$@" +} + +m_central_as () { + podman exec ovn-central "$@" +} + +check_fake_multinode_setup() { + check m_as ovn-central ovn-nbctl --wait=hv sync + AT_CHECK([m_as ovn-chassis-1 ovn-appctl -t ovn-controller version], [0], [ignore]) + AT_CHECK([m_as ovn-chassis-2 ovn-appctl -t ovn-controller version], [0], [ignore]) + AT_CHECK([m_as ovn-gw-1 ovn-appctl -t ovn-controller version], [0], [ignore]) +} + +cleanup_multinode_resources() { + m_as ovn-central rm -f /etc/ovn/ovnnb_db.db + m_as ovn-central /usr/share/ovn/scripts/ovn-ctl restart_northd + check m_as ovn-central ovn-nbctl --wait=hv sync + for c in ovn-chassis-1 ovn-chassis-2 ovn-gw-1 + do + m_as $c ovs-vsctl del-br br-int + m_as $c ip --all netns delete + done +} + +multinode_nbctl () { + m_as ovn-central ovn-nbctl "$@" +} + +# count_rows TABLE [CONDITION...] +# +# Prints the number of rows in TABLE (that satisfy CONDITION). +# Uses the southbound db by default; set DB=nb for the northbound database. +m_count_rows() { + local db=$(parse_db $1) table=$(parse_table $1); shift + m_central_as ovn-${db}ctl --format=table --no-headings find $table "$@" | wc -l +} + +# check_row_count [DATABASE:]TABLE COUNT [CONDITION...] +# +# Checks that TABLE contains COUNT rows (that satisfy CONDITION). +# The default DATABASE is "sb". +m_check_row_count() { + local db=$(parse_db $1) table=$(parse_table $1); shift + local count=$1; shift + local found=$(m_count_rows $c $db:$table "$@") + echo + echo "Checking for $count rows in $db $table${1+ with $*}... found $found" + if test "$count" != "$found"; then + m_central_as ovn-${db}ctl list $table + AT_FAIL_IF([:]) + fi +} + +# wait_row_count [DATABASE:]TABLE COUNT [CONDITION...] +# +# Waits until TABLE contains COUNT rows (that satisfy CONDITION). +# The default DATABASE is "sb". +m_wait_row_count() { + local db=$(parse_db $1) table=$(parse_table $1); shift + local count=$1; shift + local a=$1 b=$2 c=$3 d=$4 e=$5 + echo "Waiting until $count rows in $db $table${1+ with $*}..." + OVS_WAIT_UNTIL([test $count = $(m_count_rows $db:$table $a $b $c $d $e)],[ + echo "$db table $table has the following rows. $(m_count_rows $db:$table $a $b $c $d $e) rows match instead of expected $count:" + m_central_as ovn-${db}ctl list $table]) +} + +# multinode_wait_column EXPECTED [DATABASE:]TABLE [COLUMN [CONDITION...]] +# +# Wait until all of the values of COLUMN in the rows of TABLE (that +# satisfy CONDITION) equal EXPECTED (ignoring order). +# +# The default DATABASE is "sb". +# +# COLUMN defaults to _uuid if unspecified. +m_wait_column() { + local expected=$(for d in $1; do echo $d; done | sort) + local db=$(parse_db $2) table=$(parse_table $2) column=${3-_uuid}; shift; shift; shift + local a=$1 b=$2 c=$3 d=$4 e=$5 + + echo + echo "Waiting until $column in $db $table${1+ with $*} is $expected..." + OVS_WAIT_UNTIL([ + found=$(m_central_as ovn-${db}ctl --bare --columns $column find $table $a $b $c $d $e) + found=$(for d in $found; do echo $d; done | sort) + test "$expected" = "$found" + ], [ + echo "$column in $db table $table has value $found, from the following rows:" + m_central_as ovn-${db}ctl list $table]) +} + +# fetch_column [DATABASE:]TABLE COLUMN [CONDITION...] +# +# Fetches and prints all the values of COLUMN in the rows of TABLE +# (that satisfy CONDITION), sorting the results lexicographically. +# The default DATABASE is "sb". +m_fetch_column() { + local db=$(parse_db $1) table=$(parse_table $1) column=${2-_uuid}; shift; shift + # Using "echo" removes spaces and newlines. + echo $(m_central_as ovn-${db}ctl --bare --columns $column find $table "$@" | sort) +} + +# check_column EXPECTED [DATABASE:]TABLE COLUMN [CONDITION...] +# +# Fetches all of the values of COLUMN in the rows of TABLE (that +# satisfy CONDITION), and compares them against EXPECTED (ignoring +# order). +# +# The default DATABASE is "sb". +m_check_column() { + local expected=$1 db=$(parse_db $2) table=$(parse_table $2) column=${3-_uuid}; shift; shift; shift + local found=$(m_central_as ovn-${db}ctl --bare --columns $column find $table "$@") + + # Sort the expected and found values. + local found=$(for d in $found; do echo $d; done | sort) + local expected=$(for d in $expected; do echo $d; done | sort) + + echo + echo "Checking values in $db $table${1+ with $*} against $expected... found $found" + if test "$found" != "$expected"; then + m_central_as ovn-${db}ctl list $table + AT_FAIL_IF([:]) + fi +} + +# wait_column EXPECTED [DATABASE:]TABLE [COLUMN [CONDITION...]] +# +# Wait until all of the values of COLUMN in the rows of TABLE (that +# satisfy CONDITION) equal EXPECTED (ignoring order). +# +# The default DATABASE is "sb". +# +# COLUMN defaults to _uuid if unspecified. +m_wait_column() { + local expected=$(for d in $1; do echo $d; done | sort) + local db=$(parse_db $2) table=$(parse_table $2) column=${3-_uuid}; shift; shift; shift + local a=$1 b=$2 c=$3 d=$4 e=$5 + + echo + echo "Waiting until $column in $db $table${1+ with $*} is $expected..." + OVS_WAIT_UNTIL([ + found=$(m_central_as ovn-${db}ctl --bare --columns $column find $table $a $b $c $d $e) + found=$(for d in $found; do echo $d; done | sort) + test "$expected" = "$found" + ], [ + echo "$column in $db table $table has value $found, from the following rows:" + m_central_as ovn-${db}ctl list $table]) +} + +# wait_for_ports_up [PORT...] +# +# With arguments, waits for specified Logical_Switch_Ports to come up. +# Without arguments, waits for all "plain" and router +# Logical_Switch_Ports to come up. +m_wait_for_ports_up() { + if test $# = 0; then + m_wait_row_count nb:Logical_Switch_Port 0 up!=true type='""' + m_wait_row_count nb:Logical_Switch_Port 0 up!=true type=router + else + for port; do + m_wait_row_count nb:Logical_Switch_Port 1 up=true name=$port + done + fi +} + +OVS_END_SHELL_HELPERS diff --git a/tests/multinode-testsuite.at b/tests/multinode-testsuite.at new file mode 100644 index 000000000..ea10b0276 --- /dev/null +++ b/tests/multinode-testsuite.at @@ -0,0 +1,27 @@ +AT_INIT + +AT_COPYRIGHT([Copyright (c) 2022 Red Hat, + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at: + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License.]) + +m4_ifdef([AT_COLOR_TESTS], [AT_COLOR_TESTS]) +AT_ARG_OPTION([rebuild], [Do not use cached versions of databases]) + +m4_include([tests/ovs-macros.at]) +m4_include([tests/ovsdb-macros.at]) +m4_include([tests/ofproto-macros.at]) +m4_include([tests/ovn-macros.at]) +m4_include([tests/system-common-macros.at]) +m4_include([tests/multinode-macros.at]) + +m4_include([tests/multinode.at]) diff --git a/tests/multinode.at b/tests/multinode.at new file mode 100644 index 000000000..754e488d6 --- /dev/null +++ b/tests/multinode.at @@ -0,0 +1,74 @@ +AT_BANNER([ovn multinode system tests using ovn-fake-multinode]) + +AT_SETUP([ovn multinode basic test]) + +# Check that ovn-fake-multinode setup is up and running +check_fake_multinode_setup + +# Delete the multinode NB and OVS resources before starting the test. +cleanup_multinode_resources + +# Test East-West switching +check multinode_nbctl ls-add sw0 +check multinode_nbctl lsp-add sw0 sw0-port1 +check multinode_nbctl lsp-set-addresses sw0-port1 "50:54:00:00:00:03 10.0.0.3 1000::3" +check multinode_nbctl lsp-add sw0 sw0-port2 +check multinode_nbctl lsp-set-addresses sw0-port2 "50:54:00:00:00:04 10.0.0.4 1000::4" + +m_as ovn-chassis-1 /data/create_fake_vm.sh sw0-port1 sw0p1 50:54:00:00:00:03 10.0.0.3 24 10.0.0.1 1000::3/64 1000::a +m_as ovn-chassis-2 /data/create_fake_vm.sh sw0-port2 sw0p2 50:54:00:00:00:04 10.0.0.4 24 10.0.0.1 1000::4/64 1000::a + +m_wait_for_ports_up + +M_NS_CHECK_EXEC([ovn-chassis-1], [sw0p1], [ping -q -c 3 -i 0.3 -w 2 10.0.0.4 | FORMAT_PING], \ +[0], [dnl +3 packets transmitted, 3 received, 0% packet loss, time 0ms +]) + +# Add ACLs to drop all traffic +check multinode_nbctl pg-add pg0 sw0-port1 sw0-port2 +check multinode_nbctl acl-add pg0 to-lport 1001 "outport == @pg0 && ip4" drop +check multinode_nbctl --wait=hv sync + +M_NS_CHECK_EXEC([ovn-chassis-1], [sw0p1], [ping -q -c 3 -i 0.3 -w 2 10.0.0.4], \ +[1], [ignore]) + +# Add ACLs to allow icmp traffic +check multinode_nbctl acl-add pg0 to-lport 1002 "outport == @pg0 && ip4 && icmp" allow-related +check multinode_nbctl --wait=hv sync + +M_NS_CHECK_EXEC([ovn-chassis-1], [sw0p1], [ping -q -c 3 -i 0.3 -w 2 10.0.0.4 | FORMAT_PING], \ +[0], [dnl +3 packets transmitted, 3 received, 0% packet loss, time 0ms +]) + + +# Create the second logical switch with one port +check multinode_nbctl ls-add sw1 +check multinode_nbctl lsp-add sw1 sw1-port1 +check multinode_nbctl lsp-set-addresses sw1-port1 "40:54:00:00:00:03 20.0.0.3 2000::3" + +# Create a logical router and attach both logical switches +check multinode_nbctl lr-add lr0 +check multinode_nbctl lrp-add lr0 lr0-sw0 00:00:00:00:ff:01 10.0.0.1/24 1000::a/64 +check multinode_nbctl lsp-add sw0 sw0-lr0 +check multinode_nbctl lsp-set-type sw0-lr0 router +check multinode_nbctl lsp-set-addresses sw0-lr0 router +check multinode_nbctl lsp-set-options sw0-lr0 router-port=lr0-sw0 + +check multinode_nbctl lrp-add lr0 lr0-sw1 00:00:00:00:ff:02 20.0.0.1/24 2000::a/64 +check multinode_nbctl lsp-add sw1 sw1-lr0 +check multinode_nbctl lsp-set-type sw1-lr0 router +check multinode_nbctl lsp-set-addresses sw1-lr0 router +check multinode_nbctl lsp-set-options sw1-lr0 router-port=lr0-sw1 + +m_as ovn-chassis-2 /data/create_fake_vm.sh sw1-port1 sw1p1 40:54:00:00:00:03 20.0.0.3 24 20.0.0.1 2000::4/64 1000::a + +m_wait_for_ports_up sw1-port1 + +M_NS_CHECK_EXEC([ovn-chassis-1], [sw0p1], [ping -q -c 3 -i 0.3 -w 2 20.0.0.3 | FORMAT_PING], \ +[0], [dnl +3 packets transmitted, 3 received, 0% packet loss, time 0ms +]) + +AT_CLEANUP