[ovs-dev,v5,09/14] northd: Use objdep mgr for lport to lflow references.

From: Numan Siddique <numans@ovn.org>

Instead of maintaing the lport to lflow references using
'struct lflow_ref_list', this patch makes use of the existing
objdep APIs.  Since objdep_mgr is not thread safe, an instance
of objdep_mgr is maintained for each 'ovn_port'.

Once we add the thread safe support to objdep APIs we can move
the objdep_mgr to 'lflow' engine date.

Using objdep APIs does come with a cost in memory.  We now
maintain an additional hmap of ovn_lflow's to look up using
uuid.  But this will be useful to handle datapath and load
balancer changes in the lflow engine node.

This patch does few more changes which are significant:
  -  Logical flows are now hashed without the logical
     datapaths.  If a logical flow is referenced by just one
     datapath, we don't rehash it.

  -  The synthetic 'hash' column of sbrec_logical_flow now
     doesn't use the logical datapath too.  This means that
     when ovn-northd is updated/upgraded and has this commit,
     all the logical flows with 'logical_datapath' column
     set will get deleted and re-added causing some disruptions.

  -  With the commit [1] which added I-P support for logical
     port changes, multiple logical flows with same match 'M'
     and actions 'A' are generated and stored without the
     dp groups, which was not the case prior to
     that patch.
     One example to generate these lflows is:
             ovn-nbctl lsp-set-addresses sw0p1 "MAC1 IP1"
             ovn-nbctl lsp-set-addresses sw1p1 "MAC1 IP1"
	     ovn-nbctl lsp-set-addresses sw2p1 "MAC1 IP1"

     Now with this patch we go back to the earlier way.  i.e
     one logical flow with logical_dp_groups set.

  -  With this patch any updates to a logical port which
     doesn't result in new logical flows will not result in
     deletion and addition of same logical flows.
     Eg.
     ovn-nbctl set logical_switch_port sw0p1 external_ids:foo=bar
     will be a no-op to the SB logical flow table.

[1] - 8bbd67891f68("northd: Incremental processing of VIF additions in 'lflow' node.")

Suggested-by: Dumitru Ceara <dceara@redhat.com>
Signed-off-by: Numan Siddique <numans@ovn.org>
---
 lib/objdep.h             |    2 +
 lib/ovn-util.c           |   11 +-
 northd/en-lflow.c        |    9 +-
 northd/inc-proc-northd.c |    5 +-
 northd/northd.c          | 1649 +++++++++++++++++++++-----------------
 northd/northd.h          |   11 +-
 northd/ovn-northd.c      |    4 +
 7 files changed, 952 insertions(+), 739 deletions(-)

Bleep bloop.  Greetings Numan Siddique, I am a robot and I have tried out your patch.
Thanks for your contribution.

I encountered some error that I wasn't expecting.  See the details below.


checkpatch:
WARNING: Line has non-spaces leading whitespace
WARNING: Line has trailing whitespace
#3073 FILE: northd/northd.c:17013:
    

Lines checked: 3478, Warnings: 2, Errors: 0


Please check this out.  If you feel there has been an error, please email aconole@redhat.com

Thanks,
0-day Robot

On Wed, Aug 9, 2023 at 10:09 PM <numans@ovn.org> wrote:
>
> From: Numan Siddique <numans@ovn.org>
>
> Instead of maintaing the lport to lflow references using
> 'struct lflow_ref_list', this patch makes use of the existing
> objdep APIs.  Since objdep_mgr is not thread safe, an instance
> of objdep_mgr is maintained for each 'ovn_port'.
>
> Once we add the thread safe support to objdep APIs we can move
> the objdep_mgr to 'lflow' engine date.
>
> Using objdep APIs does come with a cost in memory.  We now
> maintain an additional hmap of ovn_lflow's to look up using
> uuid.  But this will be useful to handle datapath and load
> balancer changes in the lflow engine node.
>
> This patch does few more changes which are significant:
>   -  Logical flows are now hashed without the logical
>      datapaths.  If a logical flow is referenced by just one
>      datapath, we don't rehash it.
>
>   -  The synthetic 'hash' column of sbrec_logical_flow now
>      doesn't use the logical datapath too.  This means that
>      when ovn-northd is updated/upgraded and has this commit,
>      all the logical flows with 'logical_datapath' column
>      set will get deleted and re-added causing some disruptions.
>
>   -  With the commit [1] which added I-P support for logical
>      port changes, multiple logical flows with same match 'M'
>      and actions 'A' are generated and stored without the
>      dp groups, which was not the case prior to
>      that patch.
>      One example to generate these lflows is:
>              ovn-nbctl lsp-set-addresses sw0p1 "MAC1 IP1"
>              ovn-nbctl lsp-set-addresses sw1p1 "MAC1 IP1"
>              ovn-nbctl lsp-set-addresses sw2p1 "MAC1 IP1"
>
>      Now with this patch we go back to the earlier way.  i.e
>      one logical flow with logical_dp_groups set.
>
>   -  With this patch any updates to a logical port which
>      doesn't result in new logical flows will not result in
>      deletion and addition of same logical flows.
>      Eg.
>      ovn-nbctl set logical_switch_port sw0p1 external_ids:foo=bar
>      will be a no-op to the SB logical flow table.
>
> [1] - 8bbd67891f68("northd: Incremental processing of VIF additions in 'lflow' node.")
>
> Suggested-by: Dumitru Ceara <dceara@redhat.com>
> Signed-off-by: Numan Siddique <numans@ovn.org>
> ---
>  lib/objdep.h             |    2 +
>  lib/ovn-util.c           |   11 +-
>  northd/en-lflow.c        |    9 +-
>  northd/inc-proc-northd.c |    5 +-
>  northd/northd.c          | 1649 +++++++++++++++++++++-----------------
>  northd/northd.h          |   11 +-
>  northd/ovn-northd.c      |    4 +
>  7 files changed, 952 insertions(+), 739 deletions(-)
>
> diff --git a/lib/objdep.h b/lib/objdep.h
> index 1ea781947c..d599128ea3 100644
> --- a/lib/objdep.h
> +++ b/lib/objdep.h
> @@ -27,6 +27,8 @@ enum objdep_type {
>      OBJDEP_TYPE_PORTBINDING,
>      OBJDEP_TYPE_MC_GROUP,
>      OBJDEP_TYPE_TEMPLATE,
> +    OBJDEP_TYPE_LPORT,
> +    OBJDEP_TYPE_LFLOW_OD,
>      OBJDEP_TYPE_MAX,
>  };
>
> diff --git a/lib/ovn-util.c b/lib/ovn-util.c
> index 080ad4c0ce..6dc2b55d6b 100644
> --- a/lib/ovn-util.c
> +++ b/lib/ovn-util.c
> @@ -626,13 +626,10 @@ ovn_pipeline_from_name(const char *pipeline)
>  uint32_t
>  sbrec_logical_flow_hash(const struct sbrec_logical_flow *lf)
>  {
> -    const struct sbrec_datapath_binding *ld = lf->logical_datapath;
> -    uint32_t hash = ovn_logical_flow_hash(lf->table_id,
> -                                          ovn_pipeline_from_name(lf->pipeline),
> -                                          lf->priority, lf->match,
> -                                          lf->actions);
> -
> -    return ld ? ovn_logical_flow_hash_datapath(&ld->header_.uuid, hash) : hash;
> +    return ovn_logical_flow_hash(lf->table_id,
> +                                 ovn_pipeline_from_name(lf->pipeline),
> +                                 lf->priority, lf->match,
> +                                 lf->actions);
>  }
>
>  uint32_t
> diff --git a/northd/en-lflow.c b/northd/en-lflow.c
> index 77e2eff056..b7538d6382 100644
> --- a/northd/en-lflow.c
> +++ b/northd/en-lflow.c
> @@ -45,6 +45,8 @@ lflow_get_input_data(struct engine_node *node,
>          EN_OVSDB_GET(engine_get_input("SB_multicast_group", node));
>      lflow_input->sbrec_igmp_group_table =
>          EN_OVSDB_GET(engine_get_input("SB_igmp_group", node));
> +    lflow_input->sbrec_logical_dp_group_table =
> +        EN_OVSDB_GET(engine_get_input("SB_logical_dp_group", node));
>
>      lflow_input->sbrec_mcast_group_by_name_dp =
>             engine_ovsdb_node_get_index(
> @@ -85,7 +87,7 @@ void en_lflow_run(struct engine_node *node, void *data)
>                      lflow_input.sbrec_bfd_table,
>                      lflow_input.lr_ports,
>                      &bfd_connections);
> -    build_lflows(eng_ctx->ovnsb_idl_txn, &lflow_input, &lflow_data->lflows);
> +    build_lflows(eng_ctx->ovnsb_idl_txn, &lflow_input, lflow_data);
>      bfd_cleanup_connections(lflow_input.nbrec_bfd_table,
>                              &bfd_connections);
>      hmap_destroy(&bfd_connections);
> @@ -96,7 +98,7 @@ void en_lflow_run(struct engine_node *node, void *data)
>
>  bool
>  lflow_northd_handler(struct engine_node *node,
> -                     void *data)
> +                     void *data OVS_UNUSED)
>  {
>      struct northd_data *northd_data = engine_get_input_data("northd", node);
>      if (!northd_data->change_tracked) {
> @@ -116,11 +118,12 @@ lflow_northd_handler(struct engine_node *node,
>
>      if (!lflow_handle_northd_ls_changes(eng_ctx->ovnsb_idl_txn,
>                                          &northd_data->tracked_ls_changes,
> -                                        &lflow_input, &lflow_data->lflows)) {
> +                                        &lflow_input, lflow_data)) {
>          return false;
>      }
>
>      engine_set_node_state(node, EN_UPDATED);
> +
>      return true;
>  }
>
> diff --git a/northd/inc-proc-northd.c b/northd/inc-proc-northd.c
> index 9dbc2ec81a..e3bc2bda7b 100644
> --- a/northd/inc-proc-northd.c
> +++ b/northd/inc-proc-northd.c
> @@ -95,7 +95,8 @@ static unixctl_cb_func chassis_features_list;
>      SB_NODE(bfd, "bfd") \
>      SB_NODE(fdb, "fdb") \
>      SB_NODE(static_mac_binding, "static_mac_binding") \
> -    SB_NODE(chassis_template_var, "chassis_template_var")
> +    SB_NODE(chassis_template_var, "chassis_template_var") \
> +    SB_NODE(logical_dp_group, "logical_dp_group")
>
>  enum sb_engine_node {
>  #define SB_NODE(NAME, NAME_STR) SB_##NAME,
> @@ -206,6 +207,8 @@ void inc_proc_northd_init(struct ovsdb_idl_loop *nb,
>      engine_add_input(&en_lflow, &en_sb_logical_flow, NULL);
>      engine_add_input(&en_lflow, &en_sb_multicast_group, NULL);
>      engine_add_input(&en_lflow, &en_sb_igmp_group, NULL);
> +    engine_add_input(&en_lflow, &en_sb_logical_dp_group, NULL);
> +
>      engine_add_input(&en_lflow, &en_northd, lflow_northd_handler);
>
>      engine_add_input(&en_sync_to_sb_addr_set, &en_nb_address_set,
> diff --git a/northd/northd.c b/northd/northd.c
> index f709968502..7cc6189bc0 100644
> --- a/northd/northd.c
> +++ b/northd/northd.c
> @@ -1457,19 +1457,6 @@ struct ovn_port_routable_addresses {
>      size_t n_addrs;
>  };
>
> -/* A node that maintains link between an object (such as an ovn_port) and
> - * a lflow. */
> -struct lflow_ref_node {
> -    /* This list follows different lflows referenced by the same object. List
> -     * head is, for example, ovn_port->lflows.  */
> -    struct ovs_list lflow_list_node;
> -    /* This list follows different objects that reference the same lflow. List
> -     * head is ovn_lflow->referenced_by. */
> -    struct ovs_list ref_list_node;
> -    /* The lflow. */
> -    struct ovn_lflow *lflow;
> -};
> -
>  /* A logical switch port or logical router port.
>   *
>   * In steady state, an ovn_port points to a northbound Logical_Switch_Port
> @@ -1562,14 +1549,13 @@ struct ovn_port {
>      /* Temporarily used for traversing a list (or hmap) of ports. */
>      bool visited;
>
> -    /* List of struct lflow_ref_node that points to the lflows generated by
> -     * this ovn_port.
> +    /* objdep_mgr for the lflows generated by this ovn_port.
>       *
>       * This data is initialized and destroyed by the en_northd node, but
>       * populated and used only by the en_lflow node. Ideally this data should
> -     * be maintained as part of en_lflow's data (struct lflow_data): a hash
> -     * index from ovn_port key to lflows.  However, it would be less efficient
> -     * and more complex:
> +     * be maintained as part of en_lflow's data (struct lflow_data).
> +     * However, it would require thread safe access support to objdep APIs
> +     * and is more complex:
>       *
>       * 1. It would require an extra search (using the index) to find the
>       * lflows.
> @@ -1578,11 +1564,11 @@ struct ovn_port {
>       * lock which is obviously less efficient, or hash-based lock array which
>       * is more complex.
>       *
> -     * Adding the list here is more straightforward. The drawback is that we
> -     * need to keep in mind that this data belongs to en_lflow node, so never
> -     * access it from any other nodes.
> +     * Adding the lflow lflow_dep_mgr is more straightforward. The drawback
> +     * is that we need to keep in mind that this data belongs to
> +     * en_lflow node, so never access it from any other nodes.
>       */
> -    struct ovs_list lflows;
> +    struct objdep_mgr lflow_dep_mgr;
>  };
>
>  static bool lsp_can_be_inc_processed(const struct nbrec_logical_switch_port *);
> @@ -1671,12 +1657,12 @@ ovn_port_create(struct hmap *ports, const char *key,
>      op->l3dgw_port = op->cr_port = NULL;
>      hmap_insert(ports, &op->key_node, hash_string(op->key, 0));
>
> -    ovs_list_init(&op->lflows);
> +    objdep_mgr_init(&op->lflow_dep_mgr);
>      return op;
>  }
>
>  static void
> -ovn_port_destroy_orphan(struct ovn_port *port)
> +ovn_port_cleanup(struct ovn_port *port)
>  {
>      if (port->tunnel_key) {
>          ovs_assert(port->od);
> @@ -1686,6 +1672,8 @@ ovn_port_destroy_orphan(struct ovn_port *port)
>          destroy_lport_addresses(&port->lsp_addrs[i]);
>      }
>      free(port->lsp_addrs);
> +    port->n_lsp_addrs = 0;
> +    port->lsp_addrs = NULL;
>
>      if (port->peer) {
>          port->peer->peer = NULL;
> @@ -1695,20 +1683,23 @@ ovn_port_destroy_orphan(struct ovn_port *port)
>          destroy_lport_addresses(&port->ps_addrs[i]);
>      }
>      free(port->ps_addrs);
> +    port->ps_addrs = NULL;
> +    port->n_ps_addrs = 0;
>
>      destroy_routable_addresses(&port->routables);
>
>      destroy_lport_addresses(&port->lrp_networks);
>      destroy_lport_addresses(&port->proxy_arp_addrs);
> +}
> +
> +static void
> +ovn_port_destroy_orphan(struct ovn_port *port)
> +{
> +    ovn_port_cleanup(port);
>      free(port->json_key);
>      free(port->key);
> +    objdep_mgr_destroy(&port->lflow_dep_mgr);
>
> -    struct lflow_ref_node *l;
> -    LIST_FOR_EACH_SAFE (l, lflow_list_node, &port->lflows) {
> -        ovs_list_remove(&l->lflow_list_node);
> -        ovs_list_remove(&l->ref_list_node);
> -        free(l);
> -    }
>      free(port);
>  }
>
> @@ -4350,8 +4341,10 @@ build_lb_port_related_data(
>
>  struct ovn_dp_group {
>      unsigned long *bitmap;
> -    struct sbrec_logical_dp_group *dp_group;
> +    const struct sbrec_logical_dp_group *dp_group;
> +    struct uuid dpg_uuid;
>      struct hmap_node node;
> +    size_t refcnt;
>  };
>
>  static struct ovn_dp_group *
> @@ -4369,6 +4362,24 @@ ovn_dp_group_find(const struct hmap *dp_groups,
>      return NULL;
>  }
>
> +static inline void
> +inc_ovn_dp_group_ref(struct ovn_dp_group *dpg)
> +{
> +    dpg->refcnt++;
> +}
> +
> +static void
> +dec_ovn_dp_group_ref(struct hmap *dp_groups, struct ovn_dp_group *dpg)
> +{
> +    dpg->refcnt--;
> +
> +    if (!dpg->refcnt) {
> +        hmap_remove(dp_groups, &dpg->node);
> +        free(dpg->bitmap);
> +        free(dpg);
> +    }
> +}
> +
>  static struct sbrec_logical_dp_group *
>  ovn_sb_insert_or_update_logical_dp_group(
>                              struct ovsdb_idl_txn *ovnsb_txn,
> @@ -4384,7 +4395,9 @@ ovn_sb_insert_or_update_logical_dp_group(
>          sb[n++] = datapaths->array[index]->sb;
>      }
>      if (!dp_group) {
> -        dp_group = sbrec_logical_dp_group_insert(ovnsb_txn);
> +        struct uuid dpg_uuid = uuid_random();
> +        dp_group = sbrec_logical_dp_group_insert_persist_uuid(
> +            ovnsb_txn, &dpg_uuid);
>      }
>      sbrec_logical_dp_group_set_datapaths(
>          dp_group, (struct sbrec_datapath_binding **) sb, n);
> @@ -4393,12 +4406,23 @@ ovn_sb_insert_or_update_logical_dp_group(
>      return dp_group;
>  }
>
> -/* Given a desired bitmap, finds a datapath group in 'dp_groups'.  If it
> - * doesn't exist, creates a new one and adds it to 'dp_groups'.
> +static struct ovn_dp_group *
> +ovn_dp_group_get(struct hmap *dp_groups, size_t desired_n,
> +                 const unsigned long *desired_bitmap,
> +                 size_t bitmap_len)
> +{
> +    uint32_t hash;
> +
> +    hash = hash_int(desired_n, 0);
> +    return ovn_dp_group_find(dp_groups, desired_bitmap, bitmap_len, hash);
> +}
> +
> +/* Creates a new datapath group and adds it to 'dp_groups'.
>   * If 'sb_group' is provided, function will try to re-use this group by
> - * either taking it directly, or by modifying, if it's not already in use. */
> + * either taking it directly, or by modifying, if it's not already in use.
> + * Caller should first call ovn_dp_group_get() before calling this function. */
>  static struct ovn_dp_group *
> -ovn_dp_group_get_or_create(struct ovsdb_idl_txn *ovnsb_txn,
> +ovn_dp_group_create(struct ovsdb_idl_txn *ovnsb_txn,
>                             struct hmap *dp_groups,
>                             struct sbrec_logical_dp_group *sb_group,
>                             size_t desired_n,
> @@ -4409,13 +4433,6 @@ ovn_dp_group_get_or_create(struct ovsdb_idl_txn *ovnsb_txn,
>                             const struct ovn_datapaths *lr_datapaths)
>  {
>      struct ovn_dp_group *dpg;
> -    uint32_t hash;
> -
> -    hash = hash_int(desired_n, 0);
> -    dpg = ovn_dp_group_find(dp_groups, desired_bitmap, bitmap_len, hash);
> -    if (dpg) {
> -        return dpg;
> -    }
>
>      bool update_dp_group = false, can_modify = false;
>      unsigned long *dpg_bitmap;
> @@ -4460,11 +4477,39 @@ ovn_dp_group_get_or_create(struct ovsdb_idl_txn *ovnsb_txn,
>                              desired_bitmap,
>                              is_switch ? ls_datapaths : lr_datapaths);
>      }
> -    hmap_insert(dp_groups, &dpg->node, hash);
> +    dpg->dpg_uuid = dpg->dp_group->header_.uuid;
> +    hmap_insert(dp_groups, &dpg->node, hash_int(desired_n, 0));
>
>      return dpg;
>  }
>
> +/* Given a desired bitmap, finds a datapath group in 'dp_groups'.  If it
> + * doesn't exist, creates a new one and adds it to 'dp_groups'.
> + * If 'sb_group' is provided, function will try to re-use this group by
> + * either taking it directly, or by modifying, if it's not already in use. */
> +static struct ovn_dp_group *
> +ovn_dp_group_get_or_create(struct ovsdb_idl_txn *ovnsb_txn,
> +                           struct hmap *dp_groups,
> +                           struct sbrec_logical_dp_group *sb_group,
> +                           size_t desired_n,
> +                           const unsigned long *desired_bitmap,
> +                           size_t bitmap_len,
> +                           bool is_switch,
> +                           const struct ovn_datapaths *ls_datapaths,
> +                           const struct ovn_datapaths *lr_datapaths)
> +{
> +    struct ovn_dp_group *dpg;
> +
> +    dpg = ovn_dp_group_get(dp_groups, desired_n, desired_bitmap, bitmap_len);
> +    if (dpg) {
> +        return dpg;
> +    }
> +
> +    return ovn_dp_group_create(ovnsb_txn, dp_groups, sb_group, desired_n,
> +                               desired_bitmap, bitmap_len, is_switch,
> +                               ls_datapaths, lr_datapaths);
> +}
> +
>  struct sb_lb {
>      struct hmap_node hmap_node;
>
> @@ -5040,28 +5085,20 @@ ovn_port_find_in_datapath(struct ovn_datapath *od, const char *name)
>      return NULL;
>  }
>
> -static struct ovn_port *
> -ls_port_create(struct ovsdb_idl_txn *ovnsb_txn, struct hmap *ls_ports,
> -               const char *key, const struct nbrec_logical_switch_port *nbsp,
> -               struct ovn_datapath *od, const struct sbrec_port_binding *sb,
> -               struct ovs_list *lflows,
> -               const struct sbrec_mirror_table *sbrec_mirror_table,
> -               const struct sbrec_chassis_table *sbrec_chassis_table,
> -               struct ovsdb_idl_index *sbrec_chassis_by_name,
> -               struct ovsdb_idl_index *sbrec_chassis_by_hostname)
> +static bool
> +ls_port_init(struct ovn_port *op, struct ovsdb_idl_txn *ovnsb_txn,
> +             struct hmap *ls_ports, struct ovn_datapath *od,
> +             const struct sbrec_port_binding *sb,
> +             const struct sbrec_mirror_table *sbrec_mirror_table,
> +             const struct sbrec_chassis_table *sbrec_chassis_table,
> +             struct ovsdb_idl_index *sbrec_chassis_by_name,
> +             struct ovsdb_idl_index *sbrec_chassis_by_hostname)
>  {
> -    struct ovn_port *op = ovn_port_create(ls_ports, key, nbsp, NULL,
> -                                          NULL);
> -    parse_lsp_addrs(op);
>      op->od = od;
> -    hmap_insert(&od->ports, &op->dp_node, hmap_node_hash(&op->key_node));
> -    if (lflows) {
> -        ovs_list_splice(&op->lflows, lflows->next, lflows);
> -    }
> -
> +    parse_lsp_addrs(op);
>      /* Assign explicitly requested tunnel ids first. */
>      if (!ovn_port_assign_requested_tnl_id(sbrec_chassis_table, op)) {
> -        return NULL;
> +        return false;
>      }
>      if (sb) {
>          op->sb = sb;
> @@ -5078,14 +5115,57 @@ ls_port_create(struct ovsdb_idl_txn *ovnsb_txn, struct hmap *ls_ports,
>      }
>      /* Assign new tunnel ids where needed. */
>      if (!ovn_port_allocate_key(sbrec_chassis_table, ls_ports, op)) {
> -        return NULL;
> +        return false;
>      }
>      ovn_port_update_sbrec(ovnsb_txn, sbrec_chassis_by_name,
>                            sbrec_chassis_by_hostname, NULL, sbrec_mirror_table,
>                            op, NULL, NULL);
> +    return true;
> +}
> +
> +static struct ovn_port *
> +ls_port_create(struct ovsdb_idl_txn *ovnsb_txn, struct hmap *ls_ports,
> +               const char *key, const struct nbrec_logical_switch_port *nbsp,
> +               struct ovn_datapath *od, const struct sbrec_port_binding *sb,
> +               const struct sbrec_mirror_table *sbrec_mirror_table,
> +               const struct sbrec_chassis_table *sbrec_chassis_table,
> +               struct ovsdb_idl_index *sbrec_chassis_by_name,
> +               struct ovsdb_idl_index *sbrec_chassis_by_hostname)
> +{
> +    struct ovn_port *op = ovn_port_create(ls_ports, key, nbsp, NULL,
> +                                          NULL);
> +    hmap_insert(&od->ports, &op->dp_node, hmap_node_hash(&op->key_node));
> +    if (!ls_port_init(op, ovnsb_txn, ls_ports, od, sb,
> +                      sbrec_mirror_table, sbrec_chassis_table,
> +                      sbrec_chassis_by_name, sbrec_chassis_by_hostname)) {
> +        ovn_port_destroy(ls_ports, op);
> +        return NULL;
> +    }
> +
>      return op;
>  }
>
> +static bool
> +ls_port_reinit(struct ovn_port *op, struct ovsdb_idl_txn *ovnsb_txn,
> +                struct hmap *ls_ports,
> +                const struct nbrec_logical_switch_port *nbsp,
> +                const struct nbrec_logical_router_port *nbrp,
> +                struct ovn_datapath *od,
> +                const struct sbrec_port_binding *sb,
> +                const struct sbrec_mirror_table *sbrec_mirror_table,
> +                const struct sbrec_chassis_table *sbrec_chassis_table,
> +                struct ovsdb_idl_index *sbrec_chassis_by_name,
> +                struct ovsdb_idl_index *sbrec_chassis_by_hostname)
> +{
> +    ovn_port_cleanup(op);
> +    op->sb = sb;
> +    ovn_port_set_nb(op, nbsp, nbrp);
> +    op->l3dgw_port = op->cr_port = NULL;
> +    return ls_port_init(op, ovnsb_txn, ls_ports, od, sb,
> +                        sbrec_mirror_table, sbrec_chassis_table,
> +                        sbrec_chassis_by_name, sbrec_chassis_by_hostname);
> +}
> +
>  /* Returns true if the logical switch has changes which can be
>   * incrementally handled.
>   * Presently supports i-p for the below changes:
> @@ -5243,7 +5323,7 @@ ls_handle_lsp_changes(struct ovsdb_idl_txn *ovnsb_idl_txn,
>                  goto fail;
>              }
>              op = ls_port_create(ovnsb_idl_txn, &nd->ls_ports,
> -                                new_nbsp->name, new_nbsp, od, NULL, NULL,
> +                                new_nbsp->name, new_nbsp, od, NULL,
>                                  ni->sbrec_mirror_table,
>                                  ni->sbrec_chassis_table,
>                                  ni->sbrec_chassis_by_name,
> @@ -5275,17 +5355,12 @@ ls_handle_lsp_changes(struct ovsdb_idl_txn *ovnsb_idl_txn,
>                  op->visited = true;
>                  continue;
>              }
> -            struct ovs_list lflows = OVS_LIST_INITIALIZER(&lflows);
> -            ovs_list_splice(&lflows, op->lflows.next, &op->lflows);
> -            ovn_port_destroy(&nd->ls_ports, op);
> -            op = ls_port_create(ovnsb_idl_txn, &nd->ls_ports,
> -                                new_nbsp->name, new_nbsp, od, sb, &lflows,
> -                                ni->sbrec_mirror_table,
> +            if (!ls_port_reinit(op, ovnsb_idl_txn, &nd->ls_ports,
> +                                new_nbsp, NULL,
> +                                od, sb, ni->sbrec_mirror_table,
>                                  ni->sbrec_chassis_table,
>                                  ni->sbrec_chassis_by_name,
> -                                ni->sbrec_chassis_by_hostname);
> -            ovs_assert(ovs_list_is_empty(&lflows));
> -            if (!op) {
> +                                ni->sbrec_chassis_by_hostname)) {
>                  goto fail;
>              }
>              ovs_list_push_back(&ls_change->updated_ports, &op->list);
> @@ -6166,6 +6241,7 @@ ovn_igmp_group_destroy(struct hmap *igmp_groups,
>
>  struct ovn_lflow {
>      struct hmap_node hmap_node;
> +    struct hmap_node hash_node;
>      struct ovs_list list_node;   /* For temporary list of lflows. Don't remove
>                                      at destroy. */
>
> @@ -6182,15 +6258,14 @@ struct ovn_lflow {
>                                    * 'dpg_bitmap'. */
>      struct ovn_dp_group *dpg;    /* Link to unique Sb datapath group. */
>
> -    struct ovs_list referenced_by;  /* List of struct lflow_ref_node. */
>      const char *where;
>
>      struct uuid sb_uuid;         /* SB DB row uuid, specified by northd. */
> +    struct uuid lflow_uuid;
>  };
>
> -static void ovn_lflow_destroy(struct hmap *lflows, struct ovn_lflow *lflow);
> +static void ovn_lflow_destroy(struct lflow_data *, struct ovn_lflow *lflow);
>  static struct ovn_lflow *ovn_lflow_find(const struct hmap *lflows,
> -                                        const struct ovn_datapath *od,
>                                          enum ovn_stage stage,
>                                          uint16_t priority, const char *match,
>                                          const char *actions,
> @@ -6206,12 +6281,11 @@ ovn_lflow_hint(const struct ovsdb_idl_row *row)
>  }
>
>  static bool
> -ovn_lflow_equal(const struct ovn_lflow *a, const struct ovn_datapath *od,
> -                enum ovn_stage stage, uint16_t priority, const char *match,
> +ovn_lflow_equal(const struct ovn_lflow *a, enum ovn_stage stage,
> +                uint16_t priority, const char *match,
>                  const char *actions, const char *ctrl_meter)
>  {
> -    return (a->od == od
> -            && a->stage == stage
> +    return (a->stage == stage
>              && a->priority == priority
>              && !strcmp(a->match, match)
>              && !strcmp(a->actions, actions)
> @@ -6229,10 +6303,9 @@ static void
>  ovn_lflow_init(struct ovn_lflow *lflow, struct ovn_datapath *od,
>                 size_t dp_bitmap_len, enum ovn_stage stage, uint16_t priority,
>                 char *match, char *actions, char *io_port, char *ctrl_meter,
> -               char *stage_hint, const char *where)
> +               char *stage_hint, const char *where, uint32_t hash)
>  {
>      ovs_list_init(&lflow->list_node);
> -    ovs_list_init(&lflow->referenced_by);
>      lflow->dpg_bitmap = bitmap_allocate(dp_bitmap_len);
>      lflow->od = od;
>      lflow->stage = stage;
> @@ -6245,6 +6318,8 @@ ovn_lflow_init(struct ovn_lflow *lflow, struct ovn_datapath *od,
>      lflow->dpg = NULL;
>      lflow->where = where;
>      lflow->sb_uuid = UUID_ZERO;
> +    lflow->lflow_uuid = uuid_random();
> +    lflow->lflow_uuid.parts[0] = hash;
>  }
>
>  /* The lflow_hash_lock is a mutex array that protects updates to the shared
> @@ -6360,32 +6435,10 @@ ovn_dp_group_add_with_reference(struct ovn_lflow *lflow_ref,
>      }
>  }
>
> -/* This global variable collects the lflows generated by do_ovn_lflow_add().
> - * start_collecting_lflows() will enable the lflow collection and the calls to
> - * do_ovn_lflow_add (or the macros ovn_lflow_add_...) will add generated lflows
> - * to the list. end_collecting_lflows() will disable it. */
> -static thread_local struct ovs_list collected_lflows;
> -static thread_local bool collecting_lflows = false;
> -
> -static void
> -start_collecting_lflows(void)
> -{
> -    ovs_assert(!collecting_lflows);
> -    ovs_list_init(&collected_lflows);
> -    collecting_lflows = true;
> -}
> -
> -static void
> -end_collecting_lflows(void)
> -{
> -    ovs_assert(collecting_lflows);
> -    collecting_lflows = false;
> -}
> -
>  /* Adds a row with the specified contents to the Logical_Flow table.
>   * Version to use when hash bucket locking is NOT required. */
> -static void
> -do_ovn_lflow_add(struct hmap *lflow_map, const struct ovn_datapath *od,
> +static struct ovn_lflow *
> +do_ovn_lflow_add(struct lflow_data *lflow_data, const struct ovn_datapath *od,
>                   const unsigned long *dp_bitmap, size_t dp_bitmap_len,
>                   uint32_t hash, enum ovn_stage stage, uint16_t priority,
>                   const char *match, const char *actions, const char *io_port,
> @@ -6393,24 +6446,18 @@ do_ovn_lflow_add(struct hmap *lflow_map, const struct ovn_datapath *od,
>                   const char *where, const char *ctrl_meter)
>      OVS_REQUIRES(fake_hash_mutex)
>  {
> -
>      struct ovn_lflow *old_lflow;
>      struct ovn_lflow *lflow;
>
>      size_t bitmap_len = od ? ods_size(od->datapaths) : dp_bitmap_len;
>      ovs_assert(bitmap_len);
>
> -    if (collecting_lflows) {
> -        ovs_assert(od);
> -        ovs_assert(!dp_bitmap);
> -    } else {
> -        old_lflow = ovn_lflow_find(lflow_map, NULL, stage, priority, match,
> -                                   actions, ctrl_meter, hash);
> -        if (old_lflow) {
> -            ovn_dp_group_add_with_reference(old_lflow, od, dp_bitmap,
> -                                            bitmap_len);
> -            return;
> -        }
> +    old_lflow = ovn_lflow_find(&lflow_data->lflows_match_map, stage,
> +                               priority, match, actions, ctrl_meter, hash);
> +    if (old_lflow) {
> +        ovn_dp_group_add_with_reference(old_lflow, od, dp_bitmap,
> +                                        bitmap_len);
> +        return old_lflow;
>      }
>
>      lflow = xmalloc(sizeof *lflow);
> @@ -6421,25 +6468,27 @@ do_ovn_lflow_add(struct hmap *lflow_map, const struct ovn_datapath *od,
>                     xstrdup(match), xstrdup(actions),
>                     io_port ? xstrdup(io_port) : NULL,
>                     nullable_xstrdup(ctrl_meter),
> -                   ovn_lflow_hint(stage_hint), where);
> +                   ovn_lflow_hint(stage_hint), where, hash);
>
>      ovn_dp_group_add_with_reference(lflow, od, dp_bitmap, bitmap_len);
>
>      if (parallelization_state != STATE_USE_PARALLELIZATION) {
> -        hmap_insert(lflow_map, &lflow->hmap_node, hash);
> +        hmap_insert(&lflow_data->lflows_match_map, &lflow->hmap_node, hash);
> +        hmap_insert(&lflow_data->lflows_hash_map, &lflow->hash_node, hash);
>      } else {
> -        hmap_insert_fast(lflow_map, &lflow->hmap_node, hash);
> +        hmap_insert_fast(&lflow_data->lflows_match_map, &lflow->hmap_node,
> +                         hash);
> +        hmap_insert_fast(&lflow_data->lflows_hash_map, &lflow->hash_node,
> +                         hash);
>          thread_lflow_counter++;
>      }
>
> -    if (collecting_lflows) {
> -        ovs_list_insert(&collected_lflows, &lflow->list_node);
> -    }
> +    return lflow;
>  }
>
>  /* Adds a row with the specified contents to the Logical_Flow table. */
>  static void
> -ovn_lflow_add_at(struct hmap *lflow_map, const struct ovn_datapath *od,
> +ovn_lflow_add_at(struct lflow_data *lflow_data, const struct ovn_datapath *od,
>                   const unsigned long *dp_bitmap, size_t dp_bitmap_len,
>                   enum ovn_stage stage, uint16_t priority,
>                   const char *match, const char *actions, const char *io_port,
> @@ -6458,45 +6507,91 @@ ovn_lflow_add_at(struct hmap *lflow_map, const struct ovn_datapath *od,
>                                   priority, match,
>                                   actions);
>
> -    hash_lock = lflow_hash_lock(lflow_map, hash);
> -    do_ovn_lflow_add(lflow_map, od, dp_bitmap, dp_bitmap_len, hash, stage,
> +    hash_lock = lflow_hash_lock(&lflow_data->lflows_match_map, hash);
> +    do_ovn_lflow_add(lflow_data, od, dp_bitmap, dp_bitmap_len, hash, stage,
>                       priority, match, actions, io_port, stage_hint, where,
>                       ctrl_meter);
>      lflow_hash_unlock(hash_lock);
>  }
>
> +/* Adds a row with the specified contents to the Logical_Flow table. */
>  static void
> -__ovn_lflow_add_default_drop(struct hmap *lflow_map,
> +ovn_lflow_add_objdep_ref(struct lflow_data *lflow_data,
> +                         const struct ovn_datapath *od,
> +                         const unsigned long *dp_bitmap, size_t dp_bitmap_len,
> +                         enum ovn_stage stage, uint16_t priority,
> +                         const char *match, const char *actions,
> +                         const char *io_port, const char *ctrl_meter,
> +                         const struct ovsdb_idl_row *stage_hint,
> +                         const char *where, struct objdep_mgr *lflow_dep_mgr,
> +                         enum objdep_type objdep_type,
> +                         const char *res_name,
> +                         const struct ovn_datapath *res_od)
> +    OVS_EXCLUDED(fake_hash_mutex)
> +{
> +    struct ovs_mutex *hash_lock;
> +    uint32_t hash;
> +
> +    ovs_assert(lflow_dep_mgr);
> +    ovs_assert(res_name);
> +    ovs_assert(!od ||
> +               ovn_stage_to_datapath_type(stage) == ovn_datapath_get_type(od));
> +
> +    hash = ovn_logical_flow_hash(ovn_stage_get_table(stage),
> +                                 ovn_stage_get_pipeline(stage),
> +                                 priority, match,
> +                                 actions);
> +
> +    hash_lock = lflow_hash_lock(&lflow_data->lflows_match_map, hash);
> +    struct ovn_lflow *lflow =
> +        do_ovn_lflow_add(lflow_data, od, dp_bitmap, dp_bitmap_len, hash, stage,
> +                         priority, match, actions, io_port, stage_hint, where,
> +                         ctrl_meter);
> +
> +    objdep_mgr_add(lflow_dep_mgr, objdep_type, res_name, &lflow->lflow_uuid);
> +
> +    if (res_od && res_od != od) {
> +        char uuid_s[UUID_LEN + 1];
> +        sprintf(uuid_s, UUID_FMT, UUID_ARGS(&lflow->lflow_uuid));
> +
> +        struct uuid u = UUID_ZERO;
> +        u.parts[0] = od->index;
> +        objdep_mgr_add(lflow_dep_mgr, OBJDEP_TYPE_LFLOW_OD, uuid_s, &u);
> +    }
> +    lflow_hash_unlock(hash_lock);
> +}
> +
> +static void
> +__ovn_lflow_add_default_drop(struct lflow_data *lflow_data,
>                               struct ovn_datapath *od,
>                               enum ovn_stage stage,
>                               const char *where)
>  {
> -        ovn_lflow_add_at(lflow_map, od, NULL, 0, stage, 0, "1",
> -                         debug_drop_action(),
> -                         NULL, NULL, NULL, where );
> +    ovn_lflow_add_at(lflow_data, od, NULL, 0, stage, 0, "1",
> +                     debug_drop_action(), NULL, NULL, NULL, where );
>  }
>
>  /* Adds a row with the specified contents to the Logical_Flow table. */
> -#define ovn_lflow_add_with_hint__(LFLOW_MAP, OD, STAGE, PRIORITY, MATCH, \
> +#define ovn_lflow_add_with_hint__(LFLOW_DATA, OD, STAGE, PRIORITY, MATCH, \
>                                    ACTIONS, IN_OUT_PORT, CTRL_METER, \
>                                    STAGE_HINT) \
> -    ovn_lflow_add_at(LFLOW_MAP, OD, NULL, 0, STAGE, PRIORITY, MATCH, ACTIONS, \
> +    ovn_lflow_add_at(LFLOW_DATA, OD, NULL, 0, STAGE, PRIORITY, MATCH, ACTIONS,\
>                       IN_OUT_PORT, CTRL_METER, STAGE_HINT, OVS_SOURCE_LOCATOR)
>
> -#define ovn_lflow_add_with_hint(LFLOW_MAP, OD, STAGE, PRIORITY, MATCH, \
> +#define ovn_lflow_add_with_hint(LFLOW_DATA, OD, STAGE, PRIORITY, MATCH, \
>                                  ACTIONS, STAGE_HINT) \
> -    ovn_lflow_add_at(LFLOW_MAP, OD, NULL, 0, STAGE, PRIORITY, MATCH, ACTIONS, \
> +    ovn_lflow_add_at(LFLOW_DATA, OD, NULL, 0, STAGE, PRIORITY, MATCH, ACTIONS,\
>                       NULL, NULL, STAGE_HINT, OVS_SOURCE_LOCATOR)
>
> -#define ovn_lflow_add_with_dp_group(LFLOW_MAP, DP_BITMAP, DP_BITMAP_LEN, \
> +#define ovn_lflow_add_with_dp_group(LFLOW_DATA, DP_BITMAP, DP_BITMAP_LEN, \
>                                      STAGE, PRIORITY, MATCH, ACTIONS, \
>                                      STAGE_HINT) \
> -    ovn_lflow_add_at(LFLOW_MAP, NULL, DP_BITMAP, DP_BITMAP_LEN, STAGE, \
> +    ovn_lflow_add_at(LFLOW_DATA, NULL, DP_BITMAP, DP_BITMAP_LEN, STAGE, \
>                       PRIORITY, MATCH, ACTIONS, NULL, NULL, STAGE_HINT, \
>                       OVS_SOURCE_LOCATOR)
>
> -#define ovn_lflow_add_default_drop(LFLOW_MAP, OD, STAGE)                    \
> -    __ovn_lflow_add_default_drop(LFLOW_MAP, OD, STAGE, OVS_SOURCE_LOCATOR)
> +#define ovn_lflow_add_default_drop(LFLOW_DATA, OD, STAGE)                    \
> +    __ovn_lflow_add_default_drop(LFLOW_DATA, OD, STAGE, OVS_SOURCE_LOCATOR)
>
>
>  /* This macro is similar to ovn_lflow_add_with_hint, except that it requires
> @@ -6509,30 +6604,40 @@ __ovn_lflow_add_default_drop(struct hmap *lflow_map,
>   * - For egress pipeline, the lport that is used to match "outport".
>   *
>   * For now, only LS pipelines should use this macro.  */
> -#define ovn_lflow_add_with_lport_and_hint(LFLOW_MAP, OD, STAGE, PRIORITY, \
> +#define ovn_lflow_add_with_lport_and_hint(LFLOW_DATA, OD, STAGE, PRIORITY, \
>                                            MATCH, ACTIONS, IN_OUT_PORT, \
>                                            STAGE_HINT) \
> -    ovn_lflow_add_at(LFLOW_MAP, OD, NULL, 0, STAGE, PRIORITY, MATCH, ACTIONS, \
> +    ovn_lflow_add_at(LFLOW_DATA, OD, NULL, 0, STAGE, PRIORITY, MATCH, ACTIONS,\
>                       IN_OUT_PORT, NULL, STAGE_HINT, OVS_SOURCE_LOCATOR)
>
> -#define ovn_lflow_add(LFLOW_MAP, OD, STAGE, PRIORITY, MATCH, ACTIONS) \
> -    ovn_lflow_add_at(LFLOW_MAP, OD, NULL, 0, STAGE, PRIORITY, MATCH, ACTIONS, \
> +#define ovn_lflow_add(LFLOW_DATA, OD, STAGE, PRIORITY, MATCH, ACTIONS) \
> +    ovn_lflow_add_at(LFLOW_DATA, OD, NULL, 0, STAGE, PRIORITY, MATCH, ACTIONS,\
>                       NULL, NULL, NULL, OVS_SOURCE_LOCATOR)
>
> -#define ovn_lflow_metered(LFLOW_MAP, OD, STAGE, PRIORITY, MATCH, ACTIONS, \
> +#define ovn_lflow_metered(LFLOW_DATA, OD, STAGE, PRIORITY, MATCH, ACTIONS, \
>                            CTRL_METER) \
> -    ovn_lflow_add_with_hint__(LFLOW_MAP, OD, STAGE, PRIORITY, MATCH, \
> +    ovn_lflow_add_with_hint__(LFLOW_DATA, OD, STAGE, PRIORITY, MATCH, \
>                                ACTIONS, NULL, CTRL_METER, NULL)
>
> +#define ovn_lflow_add_with_lport_lflow_ref(LFLOW_DATA, OD, STAGE, PRIORITY, \
> +                                           MATCH, ACTIONS, IN_OUT_PORT, \
> +                                           CTRL_METER, STAGE_HINT, \
> +                                           LFLOW_DEP_MGR, LPORT_NAME, \
> +                                           LPORT_OD) \
> +    ovn_lflow_add_objdep_ref(LFLOW_DATA, OD, NULL, 0, STAGE, PRIORITY, MATCH, \
> +                             ACTIONS, IN_OUT_PORT, CTRL_METER, STAGE_HINT, \
> +                             OVS_SOURCE_LOCATOR, LFLOW_DEP_MGR, \
> +                             OBJDEP_TYPE_LPORT, LPORT_NAME, LPORT_OD)
> +
>  static struct ovn_lflow *
> -ovn_lflow_find(const struct hmap *lflows, const struct ovn_datapath *od,
> +ovn_lflow_find(const struct hmap *lflows,
>                 enum ovn_stage stage, uint16_t priority,
>                 const char *match, const char *actions, const char *ctrl_meter,
>                 uint32_t hash)
>  {
>      struct ovn_lflow *lflow;
>      HMAP_FOR_EACH_WITH_HASH (lflow, hmap_node, hash, lflows) {
> -        if (ovn_lflow_equal(lflow, od, stage, priority, match, actions,
> +        if (ovn_lflow_equal(lflow, stage, priority, match, actions,
>                              ctrl_meter)) {
>              return lflow;
>          }
> @@ -6540,12 +6645,27 @@ ovn_lflow_find(const struct hmap *lflows, const struct ovn_datapath *od,
>      return NULL;
>  }
>
> +static struct ovn_lflow *
> +ovn_lflow_uuid_find(const struct hmap *lflows_hash_map,
> +                    const struct uuid *lflow_uuid)
> +{
> +    uint32_t hash = lflow_uuid->parts[0];
> +    struct ovn_lflow *lflow;
> +    HMAP_FOR_EACH_WITH_HASH (lflow, hash_node, hash, lflows_hash_map) {
> +        if (uuid_equals(&lflow->lflow_uuid, lflow_uuid)) {
> +            return lflow;
> +        }
> +    }
> +    return NULL;
> +}
> +
>  static void
> -ovn_lflow_destroy(struct hmap *lflows, struct ovn_lflow *lflow)
> +ovn_lflow_destroy(struct lflow_data *lflow_data, struct ovn_lflow *lflow)
>  {
>      if (lflow) {
> -        if (lflows) {
> -            hmap_remove(lflows, &lflow->hmap_node);
> +        if (lflow_data) {
> +            hmap_remove(&lflow_data->lflows_match_map, &lflow->hmap_node);
> +            hmap_remove(&lflow_data->lflows_hash_map, &lflow->hash_node);
>          }
>          bitmap_free(lflow->dpg_bitmap);
>          free(lflow->match);
> @@ -6553,28 +6673,10 @@ ovn_lflow_destroy(struct hmap *lflows, struct ovn_lflow *lflow)
>          free(lflow->io_port);
>          free(lflow->stage_hint);
>          free(lflow->ctrl_meter);
> -        struct lflow_ref_node *l;
> -        LIST_FOR_EACH_SAFE (l, ref_list_node, &lflow->referenced_by) {
> -            ovs_list_remove(&l->lflow_list_node);
> -            ovs_list_remove(&l->ref_list_node);
> -            free(l);
> -        }
>          free(lflow);
>      }
>  }
>
> -static void
> -link_ovn_port_to_lflows(struct ovn_port *op, struct ovs_list *lflows)
> -{
> -    struct ovn_lflow *f;
> -    LIST_FOR_EACH (f, list_node, lflows) {
> -        struct lflow_ref_node *lfrn = xmalloc(sizeof *lfrn);
> -        lfrn->lflow = f;
> -        ovs_list_insert(&op->lflows, &lfrn->lflow_list_node);
> -        ovs_list_insert(&f->referenced_by, &lfrn->ref_list_node);
> -    }
> -}
> -
>  static bool
>  build_dhcpv4_action(struct ovn_port *op, ovs_be32 offer_ip,
>                      struct ds *options_action, struct ds *response_action,
> @@ -6878,8 +6980,9 @@ ls_get_acl_flags(struct ovn_datapath *od)
>   * build_lswitch_lflows_admission_control() handles the port security.
>   */
>  static void
> -build_lswitch_port_sec_op(struct ovn_port *op, struct hmap *lflows,
> -                                struct ds *actions, struct ds *match)
> +build_lswitch_port_sec_op(struct ovn_port *op, struct lflow_data *lflows,
> +                          struct ds *actions, struct ds *match,
> +                          struct objdep_mgr *lflow_dep_mgr)
>  {
>      ovs_assert(op->nbsp);
>
> @@ -6892,16 +6995,18 @@ build_lswitch_port_sec_op(struct ovn_port *op, struct hmap *lflows,
>      ds_put_format(match, "inport == %s", op->json_key);
>      if (!lsp_is_enabled(op->nbsp)) {
>          /* Drop packets from disabled logical ports. */
> -        ovn_lflow_add_with_lport_and_hint(
> +        ovn_lflow_add_with_lport_lflow_ref(
>              lflows, op->od, S_SWITCH_IN_CHECK_PORT_SEC,
>              100, ds_cstr(match), REGBIT_PORT_SEC_DROP" = 1; next;",
> -            op->key, &op->nbsp->header_);
> +            op->key, NULL, &op->nbsp->header_, lflow_dep_mgr, op->key,
> +            op->od);
>
>          ds_clear(match);
>          ds_put_format(match, "outport == %s", op->json_key);
> -        ovn_lflow_add_with_lport_and_hint(
> +        ovn_lflow_add_with_lport_lflow_ref(
>              lflows, op->od, S_SWITCH_IN_L2_UNKNOWN, 50, ds_cstr(match),
> -            debug_drop_action(), op->key, &op->nbsp->header_);
> +            debug_drop_action(), op->key, NULL, &op->nbsp->header_,
> +            lflow_dep_mgr, op->key, op->od);
>          return;
>      }
>
> @@ -6914,17 +7019,19 @@ build_lswitch_port_sec_op(struct ovn_port *op, struct hmap *lflows,
>          ds_put_format(actions, REGBIT_FROM_RAMP" = 1; ");
>          ds_put_format(actions, "next(pipeline=ingress, table=%d);",
>                        ovn_stage_get_table(S_SWITCH_IN_HAIRPIN));
> -        ovn_lflow_add_with_lport_and_hint(lflows, op->od,
> -                                          S_SWITCH_IN_CHECK_PORT_SEC, 70,
> -                                          ds_cstr(match), ds_cstr(actions),
> -                                          op->key, &op->nbsp->header_);
> +        ovn_lflow_add_with_lport_lflow_ref(lflows, op->od,
> +                                           S_SWITCH_IN_CHECK_PORT_SEC, 70,
> +                                           ds_cstr(match), ds_cstr(actions),
> +                                           op->key, NULL, &op->nbsp->header_,
> +                                           lflow_dep_mgr, op->key, op->od);
>      } else if (queue_id) {
>          ds_put_cstr(actions,
>                      REGBIT_PORT_SEC_DROP" = check_in_port_sec(); next;");
> -        ovn_lflow_add_with_lport_and_hint(lflows, op->od,
> -                                          S_SWITCH_IN_CHECK_PORT_SEC, 70,
> -                                          ds_cstr(match), ds_cstr(actions),
> -                                          op->key, &op->nbsp->header_);
> +        ovn_lflow_add_with_lport_lflow_ref(lflows, op->od,
> +                                           S_SWITCH_IN_CHECK_PORT_SEC, 70,
> +                                           ds_cstr(match), ds_cstr(actions),
> +                                           op->key, NULL, &op->nbsp->header_,
> +                                           lflow_dep_mgr, op->key, op->od);
>
>          if (!lsp_is_localnet(op->nbsp) && !op->od->n_localnet_ports) {
>              return;
> @@ -6936,27 +7043,32 @@ build_lswitch_port_sec_op(struct ovn_port *op, struct hmap *lflows,
>          ds_clear(match);
>          if (lsp_is_localnet(op->nbsp)) {
>              ds_put_format(match, "outport == %s", op->json_key);
> -            ovn_lflow_add_with_lport_and_hint(lflows, op->od,
> -                                              S_SWITCH_OUT_APPLY_PORT_SEC, 100,
> -                                              ds_cstr(match), ds_cstr(actions),
> -                                              op->key, &op->nbsp->header_);
> +            ovn_lflow_add_with_lport_lflow_ref(lflows, op->od,
> +                                               S_SWITCH_OUT_APPLY_PORT_SEC,
> +                                               100, ds_cstr(match),
> +                                               ds_cstr(actions),
> +                                               op->key, NULL,
> +                                               &op->nbsp->header_,
> +                                               lflow_dep_mgr, op->key, op->od);
>          } else if (op->od->n_localnet_ports) {
>              ds_put_format(match, "outport == %s && inport == %s",
>                            op->od->localnet_ports[0]->json_key,
>                            op->json_key);
> -            ovn_lflow_add_with_lport_and_hint(lflows, op->od,
> +            ovn_lflow_add_with_lport_lflow_ref(lflows, op->od,
>                      S_SWITCH_OUT_APPLY_PORT_SEC, 110,
>                      ds_cstr(match), ds_cstr(actions),
> -                    op->od->localnet_ports[0]->key,
> -                    &op->od->localnet_ports[0]->nbsp->header_);
> +                    op->od->localnet_ports[0]->key, NULL,
> +                    &op->od->localnet_ports[0]->nbsp->header_,
> +                    lflow_dep_mgr, op->key, op->od);
>          }
>      }
>  }
>
>  static void
>  build_lswitch_learn_fdb_op(
> -        struct ovn_port *op, struct hmap *lflows,
> -        struct ds *actions, struct ds *match)
> +        struct ovn_port *op, struct lflow_data *lflows,
> +        struct ds *actions, struct ds *match,
> +        struct objdep_mgr *lflow_dep_mgr)
>  {
>      ovs_assert(op->nbsp);
>
> @@ -6967,24 +7079,26 @@ build_lswitch_learn_fdb_op(
>          ds_put_format(match, "inport == %s", op->json_key);
>          ds_put_format(actions, REGBIT_LKUP_FDB
>                        " = lookup_fdb(inport, eth.src); next;");
> -        ovn_lflow_add_with_lport_and_hint(lflows, op->od,
> -                                          S_SWITCH_IN_LOOKUP_FDB, 100,
> -                                          ds_cstr(match), ds_cstr(actions),
> -                                          op->key, &op->nbsp->header_);
> +        ovn_lflow_add_with_lport_lflow_ref(lflows, op->od,
> +                                           S_SWITCH_IN_LOOKUP_FDB, 100,
> +                                           ds_cstr(match), ds_cstr(actions),
> +                                           op->key, NULL, &op->nbsp->header_,
> +                                           lflow_dep_mgr, op->key, op->od);
>
>          ds_put_cstr(match, " && "REGBIT_LKUP_FDB" == 0");
>          ds_clear(actions);
>          ds_put_cstr(actions, "put_fdb(inport, eth.src); next;");
> -        ovn_lflow_add_with_lport_and_hint(lflows, op->od, S_SWITCH_IN_PUT_FDB,
> -                                          100, ds_cstr(match),
> -                                          ds_cstr(actions), op->key,
> -                                          &op->nbsp->header_);
> +        ovn_lflow_add_with_lport_lflow_ref(lflows, op->od, S_SWITCH_IN_PUT_FDB,
> +                                           100, ds_cstr(match),
> +                                           ds_cstr(actions), op->key, NULL,
> +                                           &op->nbsp->header_,
> +                                           lflow_dep_mgr, op->key, op->od);
>      }
>  }
>
>  static void
>  build_lswitch_learn_fdb_od(
> -        struct ovn_datapath *od, struct hmap *lflows)
> +        struct ovn_datapath *od, struct lflow_data *lflows)
>  {
>      ovs_assert(od->nbs);
>      ovn_lflow_add(lflows, od, S_SWITCH_IN_LOOKUP_FDB, 0, "1", "next;");
> @@ -6998,7 +7112,7 @@ build_lswitch_learn_fdb_od(
>   *                 (priority 100). */
>  static void
>  build_lswitch_output_port_sec_od(struct ovn_datapath *od,
> -                              struct hmap *lflows)
> +                              struct lflow_data *lflows)
>  {
>      ovs_assert(od->nbs);
>      ovn_lflow_add(lflows, od, S_SWITCH_OUT_CHECK_PORT_SEC, 100,
> @@ -7015,7 +7129,7 @@ build_lswitch_output_port_sec_od(struct ovn_datapath *od,
>  static void
>  skip_port_from_conntrack(struct ovn_datapath *od, struct ovn_port *op,
>                           enum ovn_stage in_stage, enum ovn_stage out_stage,
> -                         uint16_t priority, struct hmap *lflows)
> +                         uint16_t priority, struct lflow_data *lflows)
>  {
>      /* Can't use ct() for router ports. Consider the following configuration:
>       * lp1(10.0.0.2) on hostA--ls1--lr0--ls2--lp2(10.0.1.2) on hostB, For a
> @@ -7049,7 +7163,7 @@ skip_port_from_conntrack(struct ovn_datapath *od, struct ovn_port *op,
>  static void
>  build_stateless_filter(struct ovn_datapath *od,
>                         const struct nbrec_acl *acl,
> -                       struct hmap *lflows)
> +                       struct lflow_data *lflows)
>  {
>      const char *action = REGBIT_ACL_STATELESS" = 1; next;";
>      if (!strcmp(acl->direction, "from-lport")) {
> @@ -7070,7 +7184,7 @@ build_stateless_filter(struct ovn_datapath *od,
>  static void
>  build_stateless_filters(struct ovn_datapath *od,
>                          const struct hmap *port_groups,
> -                        struct hmap *lflows)
> +                        struct lflow_data *lflows)
>  {
>      for (size_t i = 0; i < od->nbs->n_acls; i++) {
>          const struct nbrec_acl *acl = od->nbs->acls[i];
> @@ -7094,7 +7208,7 @@ build_stateless_filters(struct ovn_datapath *od,
>
>  static void
>  build_pre_acls(struct ovn_datapath *od, const struct hmap *port_groups,
> -               struct hmap *lflows)
> +               struct lflow_data *lflows)
>  {
>      /* Ingress and Egress Pre-ACL Table (Priority 0): Packets are
>       * allowed by default. */
> @@ -7222,7 +7336,7 @@ build_empty_lb_event_flow(struct ovn_lb_vip *lb_vip,
>  static void
>  build_interconn_mcast_snoop_flows(struct ovn_datapath *od,
>                                    const struct shash *meter_groups,
> -                                  struct hmap *lflows)
> +                                  struct lflow_data *lflows)
>  {
>      struct mcast_switch_info *mcast_sw_info = &od->mcast_info.sw;
>      if (!mcast_sw_info->enabled
> @@ -7256,7 +7370,7 @@ build_interconn_mcast_snoop_flows(struct ovn_datapath *od,
>
>  static void
>  build_pre_lb(struct ovn_datapath *od, const struct shash *meter_groups,
> -             struct hmap *lflows)
> +             struct lflow_data *lflows)
>  {
>      /* Handle IGMP/MLD packets crossing AZs. */
>      build_interconn_mcast_snoop_flows(od, meter_groups, lflows);
> @@ -7348,7 +7462,7 @@ build_pre_lb(struct ovn_datapath *od, const struct shash *meter_groups,
>  static void
>  build_pre_stateful(struct ovn_datapath *od,
>                     const struct chassis_features *features,
> -                   struct hmap *lflows)
> +                   struct lflow_data *lflows)
>  {
>      /* Ingress and Egress pre-stateful Table (Priority 0): Packets are
>       * allowed by default. */
> @@ -7380,7 +7494,7 @@ build_pre_stateful(struct ovn_datapath *od,
>  static void
>  build_acl_hints(struct ovn_datapath *od,
>                  const struct chassis_features *features,
> -                struct hmap *lflows)
> +                struct lflow_data *lflows)
>  {
>      /* This stage builds hints for the IN/OUT_ACL stage. Based on various
>       * combinations of ct flags packets may hit only a subset of the logical
> @@ -7566,7 +7680,7 @@ build_acl_log(struct ds *actions, const struct nbrec_acl *acl,
>  }
>
>  static void
> -consider_acl(struct hmap *lflows, struct ovn_datapath *od,
> +consider_acl(struct lflow_data *lflows, struct ovn_datapath *od,
>               const struct nbrec_acl *acl, bool has_stateful,
>               bool ct_masked_mark, const struct shash *meter_groups,
>               struct ds *match, struct ds *actions)
> @@ -7857,7 +7971,7 @@ build_port_group_lswitches(
>  #define IPV6_CT_OMIT_MATCH "nd || nd_ra || nd_rs || mldv1 || mldv2"
>
>  static void
> -build_acl_action_lflows(struct ovn_datapath *od, struct hmap *lflows,
> +build_acl_action_lflows(struct ovn_datapath *od, struct lflow_data *lflows,
>                          const char *default_acl_action,
>                          const struct shash *meter_groups,
>                          struct ds *match,
> @@ -7932,7 +8046,7 @@ build_acl_action_lflows(struct ovn_datapath *od, struct hmap *lflows,
>  }
>
>  static void
> -build_acl_log_related_flows(struct ovn_datapath *od, struct hmap *lflows,
> +build_acl_log_related_flows(struct ovn_datapath *od, struct lflow_data *lflows,
>                              const struct nbrec_acl *acl, bool has_stateful,
>                              bool ct_masked_mark,
>                              const struct shash *meter_groups,
> @@ -8006,7 +8120,7 @@ build_acl_log_related_flows(struct ovn_datapath *od, struct hmap *lflows,
>
>  static void
>  build_acls(struct ovn_datapath *od, const struct chassis_features *features,
> -           struct hmap *lflows, const struct hmap *port_groups,
> +           struct lflow_data *lflows, const struct hmap *port_groups,
>             const struct shash *meter_groups)
>  {
>      const char *default_acl_action = default_acl_drop
> @@ -8304,7 +8418,7 @@ build_acls(struct ovn_datapath *od, const struct chassis_features *features,
>  }
>
>  static void
> -build_qos(struct ovn_datapath *od, struct hmap *lflows) {
> +build_qos(struct ovn_datapath *od, struct lflow_data *lflows) {
>      struct ds action = DS_EMPTY_INITIALIZER;
>
>      ovn_lflow_add(lflows, od, S_SWITCH_IN_QOS_MARK, 0, "1", "next;");
> @@ -8365,7 +8479,7 @@ build_qos(struct ovn_datapath *od, struct hmap *lflows) {
>  }
>
>  static void
> -build_lb_rules_pre_stateful(struct hmap *lflows,
> +build_lb_rules_pre_stateful(struct lflow_data *lflows,
>                              struct ovn_lb_datapaths *lb_dps,
>                              bool ct_lb_mark,
>                              const struct ovn_datapaths *ls_datapaths,
> @@ -8467,7 +8581,8 @@ build_lb_rules_pre_stateful(struct hmap *lflows,
>   *
>   */
>  static void
> -build_lb_affinity_lr_flows(struct hmap *lflows, const struct ovn_northd_lb *lb,
> +build_lb_affinity_lr_flows(struct lflow_data *lflows,
> +                           const struct ovn_northd_lb *lb,
>                             struct ovn_lb_vip *lb_vip, char *new_lb_match,
>                             char *lb_action, const unsigned long *dp_bitmap,
>                             const struct ovn_datapaths *lr_datapaths)
> @@ -8653,7 +8768,7 @@ build_lb_affinity_lr_flows(struct hmap *lflows, const struct ovn_northd_lb *lb,
>   *
>   */
>  static void
> -build_lb_affinity_ls_flows(struct hmap *lflows,
> +build_lb_affinity_ls_flows(struct lflow_data *lflows,
>                             struct ovn_lb_datapaths *lb_dps,
>                             struct ovn_lb_vip *lb_vip,
>                             const struct ovn_datapaths *ls_datapaths)
> @@ -8796,7 +8911,7 @@ build_lb_affinity_ls_flows(struct hmap *lflows,
>
>  static void
>  build_lswitch_lb_affinity_default_flows(struct ovn_datapath *od,
> -                                        struct hmap *lflows)
> +                                        struct lflow_data *lflows)
>  {
>      ovs_assert(od->nbs);
>      ovn_lflow_add(lflows, od, S_SWITCH_IN_LB_AFF_CHECK, 0, "1", "next;");
> @@ -8805,7 +8920,7 @@ build_lswitch_lb_affinity_default_flows(struct ovn_datapath *od,
>
>  static void
>  build_lrouter_lb_affinity_default_flows(struct ovn_datapath *od,
> -                                        struct hmap *lflows)
> +                                        struct lflow_data *lflows)
>  {
>      ovs_assert(od->nbr);
>      ovn_lflow_add(lflows, od, S_ROUTER_IN_LB_AFF_CHECK, 0, "1", "next;");
> @@ -8813,7 +8928,7 @@ build_lrouter_lb_affinity_default_flows(struct ovn_datapath *od,
>  }
>
>  static void
> -build_lb_rules(struct hmap *lflows, struct ovn_lb_datapaths *lb_dps,
> +build_lb_rules(struct lflow_data *lflows, struct ovn_lb_datapaths *lb_dps,
>                 const struct ovn_datapaths *ls_datapaths,
>                 const struct chassis_features *features, struct ds *match,
>                 struct ds *action, const struct shash *meter_groups,
> @@ -8893,7 +9008,7 @@ build_lb_rules(struct hmap *lflows, struct ovn_lb_datapaths *lb_dps,
>  static void
>  build_stateful(struct ovn_datapath *od,
>                 const struct chassis_features *features,
> -               struct hmap *lflows)
> +               struct lflow_data *lflows)
>  {
>      const char *ct_block_action = features->ct_no_masked_label
>                                    ? "ct_mark.blocked"
> @@ -8942,7 +9057,7 @@ build_stateful(struct ovn_datapath *od,
>  }
>
>  static void
> -build_lb_hairpin(struct ovn_datapath *od, struct hmap *lflows)
> +build_lb_hairpin(struct ovn_datapath *od, struct lflow_data *lflows)
>  {
>      /* Ingress Pre-Hairpin/Nat-Hairpin/Hairpin tabled (Priority 0).
>       * Packets that don't need hairpinning should continue processing.
> @@ -8999,7 +9114,7 @@ build_lb_hairpin(struct ovn_datapath *od, struct hmap *lflows)
>  }
>
>  static void
> -build_vtep_hairpin(struct ovn_datapath *od, struct hmap *lflows)
> +build_vtep_hairpin(struct ovn_datapath *od, struct lflow_data *lflows)
>  {
>      if (!od->has_vtep_lports) {
>          /* There is no need in these flows if datapath has no vtep lports. */
> @@ -9047,7 +9162,7 @@ build_vtep_hairpin(struct ovn_datapath *od, struct hmap *lflows)
>
>  /* Build logical flows for the forwarding groups */
>  static void
> -build_fwd_group_lflows(struct ovn_datapath *od, struct hmap *lflows)
> +build_fwd_group_lflows(struct ovn_datapath *od, struct lflow_data *lflows)
>  {
>      ovs_assert(od->nbs);
>      if (!od->nbs->n_forwarding_groups) {
> @@ -9225,7 +9340,10 @@ static void
>  build_lswitch_rport_arp_req_self_orig_flow(struct ovn_port *op,
>                                             uint32_t priority,
>                                             struct ovn_datapath *od,
> -                                           struct hmap *lflows)
> +                                           struct ovn_port *patch_op,
> +                                           struct lflow_data *lflows,
> +                                           const struct ovsdb_idl_row *hint,
> +                                           struct objdep_mgr *lflow_dep_mgr)
>  {
>      struct sset all_eth_addrs = SSET_INITIALIZER(&all_eth_addrs);
>      struct ds eth_src = DS_EMPTY_INITIALIZER;
> @@ -9270,8 +9388,11 @@ build_lswitch_rport_arp_req_self_orig_flow(struct ovn_port *op,
>      ds_put_format(&match,
>                    "eth.src == %s && (arp.op == 1 || rarp.op == 3 || nd_ns)",
>                    ds_cstr(&eth_src));
> -    ovn_lflow_add(lflows, od, S_SWITCH_IN_L2_LKUP, priority, ds_cstr(&match),
> -                  "outport = \""MC_FLOOD_L2"\"; output;");
> +    ovn_lflow_add_with_lport_lflow_ref(lflows, od, S_SWITCH_IN_L2_LKUP,
> +                                       priority, ds_cstr(&match),
> +                                       "outport = \""MC_FLOOD_L2"\"; output;",
> +                                       NULL, NULL, hint, lflow_dep_mgr,
> +                                       patch_op->key, patch_op->od);
>
>      sset_destroy(&all_eth_addrs);
>      ds_destroy(&eth_src);
> @@ -9339,8 +9460,9 @@ lrouter_port_ipv6_reachable(const struct ovn_port *op,
>  static void
>  build_lswitch_rport_arp_req_flow(const char *ips,
>      int addr_family, struct ovn_port *patch_op, struct ovn_datapath *od,
> -    uint32_t priority, struct hmap *lflows,
> -    const struct ovsdb_idl_row *stage_hint)
> +    uint32_t priority, struct lflow_data *lflows,
> +    const struct ovsdb_idl_row *stage_hint,
> +    struct objdep_mgr *lflow_dep_mgr)
>  {
>      struct ds match   = DS_EMPTY_INITIALIZER;
>      struct ds actions = DS_EMPTY_INITIALIZER;
> @@ -9354,14 +9476,16 @@ build_lswitch_rport_arp_req_flow(const char *ips,
>          ds_put_format(&actions, "clone {outport = %s; output; }; "
>                                  "outport = \""MC_FLOOD_L2"\"; output;",
>                        patch_op->json_key);
> -        ovn_lflow_add_with_hint(lflows, od, S_SWITCH_IN_L2_LKUP,
> +        ovn_lflow_add_with_lport_lflow_ref(lflows, od, S_SWITCH_IN_L2_LKUP,
>                                  priority, ds_cstr(&match),
> -                                ds_cstr(&actions), stage_hint);
> +                                ds_cstr(&actions), NULL, NULL, stage_hint,
> +                                lflow_dep_mgr, patch_op->key, patch_op->od);
>      } else {
>          ds_put_format(&actions, "outport = %s; output;", patch_op->json_key);
> -        ovn_lflow_add_with_hint(lflows, od, S_SWITCH_IN_L2_LKUP, priority,
> -                                ds_cstr(&match), ds_cstr(&actions),
> -                                stage_hint);
> +        ovn_lflow_add_with_lport_lflow_ref(lflows, od, S_SWITCH_IN_L2_LKUP,
> +                                priority, ds_cstr(&match), ds_cstr(&actions),
> +                                NULL, NULL, stage_hint, lflow_dep_mgr,
> +                                patch_op->key, patch_op->od);
>      }
>
>      ds_destroy(&match);
> @@ -9379,8 +9503,9 @@ static void
>  build_lswitch_rport_arp_req_flows(struct ovn_port *op,
>                                    struct ovn_datapath *sw_od,
>                                    struct ovn_port *sw_op,
> -                                  struct hmap *lflows,
> -                                  const struct ovsdb_idl_row *stage_hint)
> +                                  struct lflow_data *lflows,
> +                                  const struct ovsdb_idl_row *stage_hint,
> +                                  struct objdep_mgr *lflow_dep_mgr)
>  {
>      if (!op || !op->nbrp) {
>          return;
> @@ -9406,7 +9531,7 @@ build_lswitch_rport_arp_req_flows(struct ovn_port *op,
>              lrouter_port_ipv4_reachable(op, ipv4_addr)) {
>              build_lswitch_rport_arp_req_flow(
>                  ip_addr, AF_INET, sw_op, sw_od, 80, lflows,
> -                stage_hint);
> +                stage_hint, lflow_dep_mgr);
>          }
>      }
>      SSET_FOR_EACH (ip_addr, &op->od->lb_ips->ips_v6_reachable) {
> @@ -9419,7 +9544,7 @@ build_lswitch_rport_arp_req_flows(struct ovn_port *op,
>              lrouter_port_ipv6_reachable(op, &ipv6_addr)) {
>              build_lswitch_rport_arp_req_flow(
>                  ip_addr, AF_INET6, sw_op, sw_od, 80, lflows,
> -                stage_hint);
> +                stage_hint, lflow_dep_mgr);
>          }
>      }
>
> @@ -9442,13 +9567,13 @@ build_lswitch_rport_arp_req_flows(struct ovn_port *op,
>              if (!sset_contains(&op->od->lb_ips->ips_v6, nat->external_ip)) {
>                  build_lswitch_rport_arp_req_flow(
>                      nat->external_ip, AF_INET6, sw_op, sw_od, 80, lflows,
> -                    stage_hint);
> +                    stage_hint, lflow_dep_mgr);
>              }
>          } else {
>              if (!sset_contains(&op->od->lb_ips->ips_v4, nat->external_ip)) {
>                  build_lswitch_rport_arp_req_flow(
>                      nat->external_ip, AF_INET, sw_op, sw_od, 80, lflows,
> -                    stage_hint);
> +                    stage_hint, lflow_dep_mgr);
>              }
>          }
>      }
> @@ -9456,12 +9581,12 @@ build_lswitch_rport_arp_req_flows(struct ovn_port *op,
>      for (size_t i = 0; i < op->lrp_networks.n_ipv4_addrs; i++) {
>          build_lswitch_rport_arp_req_flow(
>              op->lrp_networks.ipv4_addrs[i].addr_s, AF_INET, sw_op, sw_od, 80,
> -            lflows, stage_hint);
> +            lflows, stage_hint, lflow_dep_mgr);
>      }
>      for (size_t i = 0; i < op->lrp_networks.n_ipv6_addrs; i++) {
>          build_lswitch_rport_arp_req_flow(
>              op->lrp_networks.ipv6_addrs[i].addr_s, AF_INET6, sw_op, sw_od, 80,
> -            lflows, stage_hint);
> +            lflows, stage_hint, lflow_dep_mgr);
>      }
>
>      /* Self originated ARP requests/RARP/ND need to be flooded as usual.
> @@ -9472,7 +9597,9 @@ build_lswitch_rport_arp_req_flows(struct ovn_port *op,
>       * Priority: 75.
>       */
>      if (sw_od->n_router_ports != sw_od->nbs->n_ports) {
> -        build_lswitch_rport_arp_req_self_orig_flow(op, 75, sw_od, lflows);
> +        build_lswitch_rport_arp_req_self_orig_flow(op, 75, sw_od, sw_op,
> +                                                   lflows, stage_hint,
> +                                                   lflow_dep_mgr);
>      }
>  }
>
> @@ -9481,7 +9608,8 @@ build_dhcpv4_options_flows(struct ovn_port *op,
>                             struct lport_addresses *lsp_addrs,
>                             struct ovn_port *inport, bool is_external,
>                             const struct shash *meter_groups,
> -                           struct hmap *lflows)
> +                           struct lflow_data *lflows,
> +                           struct objdep_mgr *lflow_dep_mgr)
>  {
>      struct ds match = DS_EMPTY_INITIALIZER;
>
> @@ -9504,7 +9632,7 @@ build_dhcpv4_options_flows(struct ovn_port *op,
>                                op->json_key);
>              }
>
> -            ovn_lflow_add_with_hint__(lflows, op->od,
> +            ovn_lflow_add_with_lport_lflow_ref(lflows, op->od,
>                                        S_SWITCH_IN_DHCP_OPTIONS, 100,
>                                        ds_cstr(&match),
>                                        ds_cstr(&options_action),
> @@ -9512,7 +9640,8 @@ build_dhcpv4_options_flows(struct ovn_port *op,
>                                        copp_meter_get(COPP_DHCPV4_OPTS,
>                                                       op->od->nbs->copp,
>                                                       meter_groups),
> -                                      &op->nbsp->dhcpv4_options->header_);
> +                                      &op->nbsp->dhcpv4_options->header_,
> +                                      lflow_dep_mgr, op->key, op->od);
>              ds_clear(&match);
>              /* Allow ip4.src = OFFER_IP and
>               * ip4.dst = {SERVER_IP, 255.255.255.255} for the below
> @@ -9532,7 +9661,7 @@ build_dhcpv4_options_flows(struct ovn_port *op,
>                                op->json_key);
>              }
>
> -            ovn_lflow_add_with_hint__(lflows, op->od,
> +            ovn_lflow_add_with_lport_lflow_ref(lflows, op->od,
>                                        S_SWITCH_IN_DHCP_OPTIONS, 100,
>                                        ds_cstr(&match),
>                                        ds_cstr(&options_action),
> @@ -9540,7 +9669,8 @@ build_dhcpv4_options_flows(struct ovn_port *op,
>                                        copp_meter_get(COPP_DHCPV4_OPTS,
>                                                       op->od->nbs->copp,
>                                                       meter_groups),
> -                                      &op->nbsp->dhcpv4_options->header_);
> +                                      &op->nbsp->dhcpv4_options->header_,
> +                                      lflow_dep_mgr, op->key, op->od);
>              ds_clear(&match);
>
>              /* If REGBIT_DHCP_OPTS_RESULT is set, it means the
> @@ -9556,10 +9686,11 @@ build_dhcpv4_options_flows(struct ovn_port *op,
>                                op->json_key);
>              }
>
> -            ovn_lflow_add_with_lport_and_hint(
> +            ovn_lflow_add_with_lport_lflow_ref(
>                  lflows, op->od, S_SWITCH_IN_DHCP_RESPONSE, 100,
>                  ds_cstr(&match), ds_cstr(&response_action), inport->key,
> -                &op->nbsp->dhcpv4_options->header_);
> +                NULL, &op->nbsp->dhcpv4_options->header_,
> +                lflow_dep_mgr, op->key, op->od);
>              ds_destroy(&options_action);
>              ds_destroy(&response_action);
>              ds_destroy(&ipv4_addr_match);
> @@ -9574,7 +9705,8 @@ build_dhcpv6_options_flows(struct ovn_port *op,
>                             struct lport_addresses *lsp_addrs,
>                             struct ovn_port *inport, bool is_external,
>                             const struct shash *meter_groups,
> -                           struct hmap *lflows)
> +                           struct lflow_data *lflows,
> +                           struct objdep_mgr *lflow_dep_mgr)
>  {
>      struct ds match = DS_EMPTY_INITIALIZER;
>
> @@ -9596,7 +9728,7 @@ build_dhcpv6_options_flows(struct ovn_port *op,
>                                op->json_key);
>              }
>
> -            ovn_lflow_add_with_hint__(lflows, op->od,
> +            ovn_lflow_add_with_lport_lflow_ref(lflows, op->od,
>                                        S_SWITCH_IN_DHCP_OPTIONS, 100,
>                                        ds_cstr(&match),
>                                        ds_cstr(&options_action),
> @@ -9604,15 +9736,17 @@ build_dhcpv6_options_flows(struct ovn_port *op,
>                                        copp_meter_get(COPP_DHCPV6_OPTS,
>                                                       op->od->nbs->copp,
>                                                       meter_groups),
> -                                      &op->nbsp->dhcpv6_options->header_);
> +                                      &op->nbsp->dhcpv6_options->header_,
> +                                      lflow_dep_mgr, op->key, op->od);
>
>              /* If REGBIT_DHCP_OPTS_RESULT is set to 1, it means the
>               * put_dhcpv6_opts action is successful */
>              ds_put_cstr(&match, " && "REGBIT_DHCP_OPTS_RESULT);
> -            ovn_lflow_add_with_lport_and_hint(
> +            ovn_lflow_add_with_lport_lflow_ref(
>                  lflows, op->od, S_SWITCH_IN_DHCP_RESPONSE, 100,
>                  ds_cstr(&match), ds_cstr(&response_action), inport->key,
> -                &op->nbsp->dhcpv6_options->header_);
> +                NULL, &op->nbsp->dhcpv6_options->header_,
> +                lflow_dep_mgr, op->key, op->od);
>              ds_destroy(&options_action);
>              ds_destroy(&response_action);
>              break;
> @@ -9624,7 +9758,8 @@ build_dhcpv6_options_flows(struct ovn_port *op,
>  static void
>  build_drop_arp_nd_flows_for_unbound_router_ports(struct ovn_port *op,
>                                                   const struct ovn_port *port,
> -                                                 struct hmap *lflows)
> +                                                 struct lflow_data *lflows,
> +                                                 struct objdep_mgr *dep_mgr)
>  {
>      struct ds match = DS_EMPTY_INITIALIZER;
>
> @@ -9641,10 +9776,10 @@ build_drop_arp_nd_flows_for_unbound_router_ports(struct ovn_port *op,
>                          port->json_key,
>                          op->lsp_addrs[i].ea_s, op->json_key,
>                          rp->lsp_addrs[k].ipv4_addrs[l].addr_s);
> -                    ovn_lflow_add_with_lport_and_hint(
> +                    ovn_lflow_add_with_lport_lflow_ref(
>                          lflows, op->od, S_SWITCH_IN_EXTERNAL_PORT, 100,
>                          ds_cstr(&match),  debug_drop_action(), port->key,
> -                        &op->nbsp->header_);
> +                        NULL, &op->nbsp->header_, dep_mgr, op->key, op->od);
>                  }
>                  for (size_t l = 0; l < rp->lsp_addrs[k].n_ipv6_addrs; l++) {
>                      ds_clear(&match);
> @@ -9657,10 +9792,10 @@ build_drop_arp_nd_flows_for_unbound_router_ports(struct ovn_port *op,
>                          rp->lsp_addrs[k].ipv6_addrs[l].addr_s,
>                          rp->lsp_addrs[k].ipv6_addrs[l].sn_addr_s,
>                          rp->lsp_addrs[k].ipv6_addrs[l].addr_s);
> -                    ovn_lflow_add_with_lport_and_hint(
> +                    ovn_lflow_add_with_lport_lflow_ref(
>                          lflows, op->od, S_SWITCH_IN_EXTERNAL_PORT, 100,
>                          ds_cstr(&match), debug_drop_action(), port->key,
> -                        &op->nbsp->header_);
> +                        NULL, &op->nbsp->header_, dep_mgr, op->key, op->od);
>                  }
>
>                  ds_clear(&match);
> @@ -9671,12 +9806,13 @@ build_drop_arp_nd_flows_for_unbound_router_ports(struct ovn_port *op,
>                      port->json_key,
>                      op->lsp_addrs[i].ea_s, rp->lsp_addrs[k].ea_s,
>                      op->json_key);
> -                ovn_lflow_add_with_lport_and_hint(lflows, op->od,
> +                ovn_lflow_add_with_lport_lflow_ref(lflows, op->od,
>                                                    S_SWITCH_IN_EXTERNAL_PORT,
>                                                    100, ds_cstr(&match),
>                                                    debug_drop_action(),
> -                                                  port->key,
> -                                                  &op->nbsp->header_);
> +                                                  port->key, NULL,
> +                                                  &op->nbsp->header_,
> +                                                  dep_mgr, op->key, op->od);
>              }
>          }
>      }
> @@ -9691,7 +9827,7 @@ is_vlan_transparent(const struct ovn_datapath *od)
>
>  static void
>  build_lswitch_lflows_l2_unknown(struct ovn_datapath *od,
> -                                struct hmap *lflows)
> +                                struct lflow_data *lflows)
>  {
>      /* Ingress table 25/26: Destination lookup for unknown MACs. */
>      if (od->has_unknown) {
> @@ -9712,7 +9848,7 @@ static void
>  build_lswitch_lflows_pre_acl_and_acl(struct ovn_datapath *od,
>                                       const struct hmap *port_groups,
>                                       const struct chassis_features *features,
> -                                     struct hmap *lflows,
> +                                     struct lflow_data *lflows,
>                                       const struct shash *meter_groups)
>  {
>      ovs_assert(od->nbs);
> @@ -9733,7 +9869,7 @@ build_lswitch_lflows_pre_acl_and_acl(struct ovn_datapath *od,
>   * 100). */
>  static void
>  build_lswitch_lflows_admission_control(struct ovn_datapath *od,
> -                                       struct hmap *lflows)
> +                                       struct lflow_data *lflows)
>  {
>      ovs_assert(od->nbs);
>      /* Logical VLANs not supported. */
> @@ -9761,8 +9897,9 @@ build_lswitch_lflows_admission_control(struct ovn_datapath *od,
>
>  static void
>  build_lswitch_arp_nd_responder_skip_local(struct ovn_port *op,
> -                                          struct hmap *lflows,
> -                                          struct ds *match)
> +                                          struct lflow_data *lflows,
> +                                          struct ds *match,
> +                                          struct objdep_mgr *lflow_dep_mgr)
>  {
>      ovs_assert(op->nbsp);
>      if (!lsp_is_localnet(op->nbsp) || op->od->has_arp_proxy_port) {
> @@ -9770,21 +9907,23 @@ build_lswitch_arp_nd_responder_skip_local(struct ovn_port *op,
>      }
>      ds_clear(match);
>      ds_put_format(match, "inport == %s", op->json_key);
> -    ovn_lflow_add_with_lport_and_hint(lflows, op->od,
> -                                      S_SWITCH_IN_ARP_ND_RSP, 100,
> -                                      ds_cstr(match), "next;", op->key,
> -                                      &op->nbsp->header_);
> +    ovn_lflow_add_with_lport_lflow_ref(lflows, op->od,
> +                                       S_SWITCH_IN_ARP_ND_RSP, 100,
> +                                       ds_cstr(match), "next;", op->key,
> +                                       NULL, &op->nbsp->header_,
> +                                       lflow_dep_mgr, op->key, op->od);
>  }
>
>  /* Ingress table 19: ARP/ND responder, reply for known IPs.
>   * (priority 50). */
>  static void
>  build_lswitch_arp_nd_responder_known_ips(struct ovn_port *op,
> -                                         struct hmap *lflows,
> +                                         struct lflow_data *lflows,
>                                           const struct hmap *ls_ports,
>                                           const struct shash *meter_groups,
>                                           struct ds *actions,
> -                                         struct ds *match)
> +                                         struct ds *match,
> +                                         struct objdep_mgr *lflow_dep_mgr)
>  {
>      ovs_assert(op->nbsp);
>      if (!strcmp(op->nbsp->type, "virtual")) {
> @@ -9861,11 +10000,12 @@ build_lswitch_arp_nd_responder_known_ips(struct ovn_port *op,
>                  "bind_vport(%s, inport); "
>                  "next;",
>                  op->json_key);
> -            ovn_lflow_add_with_lport_and_hint(lflows, op->od,
> -                                              S_SWITCH_IN_ARP_ND_RSP, 100,
> -                                              ds_cstr(match),
> -                                              ds_cstr(actions), vparent,
> -                                              &vp->nbsp->header_);
> +            ovn_lflow_add_with_lport_lflow_ref(lflows, op->od,
> +                                               S_SWITCH_IN_ARP_ND_RSP, 100,
> +                                               ds_cstr(match),
> +                                               ds_cstr(actions), vparent,
> +                                               NULL, &vp->nbsp->header_,
> +                                               lflow_dep_mgr, op->key, op->od);
>          }
>
>          free(tokstr);
> @@ -9909,11 +10049,12 @@ build_lswitch_arp_nd_responder_known_ips(struct ovn_port *op,
>                      "output;",
>                      op->lsp_addrs[i].ea_s, op->lsp_addrs[i].ea_s,
>                      op->lsp_addrs[i].ipv4_addrs[j].addr_s);
> -                ovn_lflow_add_with_hint(lflows, op->od,
> +                ovn_lflow_add_with_lport_lflow_ref(lflows, op->od,
>                                          S_SWITCH_IN_ARP_ND_RSP, 50,
>                                          ds_cstr(match),
> -                                        ds_cstr(actions),
> -                                        &op->nbsp->header_);
> +                                        ds_cstr(actions), NULL, NULL,
> +                                        &op->nbsp->header_,
> +                                        lflow_dep_mgr, op->key, op->od);
>
>                  /* Do not reply to an ARP request from the port that owns
>                   * the address (otherwise a DHCP client that ARPs to check
> @@ -9928,11 +10069,13 @@ build_lswitch_arp_nd_responder_known_ips(struct ovn_port *op,
>                   * network is not working as configured, so dropping the
>                   * request would frustrate that intent.) */
>                  ds_put_format(match, " && inport == %s", op->json_key);
> -                ovn_lflow_add_with_lport_and_hint(lflows, op->od,
> +                ovn_lflow_add_with_lport_lflow_ref(lflows, op->od,
>                                                    S_SWITCH_IN_ARP_ND_RSP,
>                                                    100, ds_cstr(match),
> -                                                  "next;", op->key,
> -                                                  &op->nbsp->header_);
> +                                                  "next;", op->key, NULL,
> +                                                  &op->nbsp->header_,
> +                                                  lflow_dep_mgr, op->key,
> +                                                  op->od);
>              }
>
>              /* For ND solicitations, we need to listen for both the
> @@ -9962,24 +10105,28 @@ build_lswitch_arp_nd_responder_known_ips(struct ovn_port *op,
>                          op->lsp_addrs[i].ipv6_addrs[j].addr_s,
>                          op->lsp_addrs[i].ipv6_addrs[j].addr_s,
>                          op->lsp_addrs[i].ea_s);
> -                ovn_lflow_add_with_hint__(lflows, op->od,
> -                                          S_SWITCH_IN_ARP_ND_RSP, 50,
> -                                          ds_cstr(match),
> -                                          ds_cstr(actions),
> -                                          NULL,
> -                                          copp_meter_get(COPP_ND_NA,
> -                                              op->od->nbs->copp,
> -                                              meter_groups),
> -                                          &op->nbsp->header_);
> +                ovn_lflow_add_with_lport_lflow_ref(lflows, op->od,
> +                                                   S_SWITCH_IN_ARP_ND_RSP, 50,
> +                                                   ds_cstr(match),
> +                                                   ds_cstr(actions),
> +                                                   NULL,
> +                                                   copp_meter_get(COPP_ND_NA,
> +                                                        op->od->nbs->copp,
> +                                                        meter_groups),
> +                                                   &op->nbsp->header_,
> +                                                   lflow_dep_mgr, op->key,
> +                                                   op->od);
>
>                  /* Do not reply to a solicitation from the port that owns
>                   * the address (otherwise DAD detection will fail). */
>                  ds_put_format(match, " && inport == %s", op->json_key);
> -                ovn_lflow_add_with_lport_and_hint(lflows, op->od,
> -                                                  S_SWITCH_IN_ARP_ND_RSP,
> -                                                  100, ds_cstr(match),
> -                                                  "next;", op->key,
> -                                                  &op->nbsp->header_);
> +                ovn_lflow_add_with_lport_lflow_ref(lflows, op->od,
> +                                                   S_SWITCH_IN_ARP_ND_RSP,
> +                                                   100, ds_cstr(match),
> +                                                   "next;", op->key, NULL,
> +                                                   &op->nbsp->header_,
> +                                                   lflow_dep_mgr, op->key,
> +                                                   op->od);
>              }
>          }
>      }
> @@ -10025,8 +10172,11 @@ build_lswitch_arp_nd_responder_known_ips(struct ovn_port *op,
>                  ea_s,
>                  ea_s);
>
> -            ovn_lflow_add_with_hint(lflows, op->od, S_SWITCH_IN_ARP_ND_RSP,
> -                30, ds_cstr(match), ds_cstr(actions), &op->nbsp->header_);
> +            ovn_lflow_add_with_lport_lflow_ref(lflows, op->od,
> +                                    S_SWITCH_IN_ARP_ND_RSP,
> +                                    30, ds_cstr(match), ds_cstr(actions),
> +                                    NULL, NULL, &op->nbsp->header_,
> +                                    lflow_dep_mgr, op->key, op->od);
>          }
>
>          /* Add IPv6 NDP responses.
> @@ -10069,7 +10219,7 @@ build_lswitch_arp_nd_responder_known_ips(struct ovn_port *op,
>                      lsp_is_router(op->nbsp) ? "nd_na_router" : "nd_na",
>                      ea_s,
>                      ea_s);
> -            ovn_lflow_add_with_hint__(lflows, op->od,
> +            ovn_lflow_add_with_lport_lflow_ref(lflows, op->od,
>                                        S_SWITCH_IN_ARP_ND_RSP, 30,
>                                        ds_cstr(match),
>                                        ds_cstr(actions),
> @@ -10077,7 +10227,8 @@ build_lswitch_arp_nd_responder_known_ips(struct ovn_port *op,
>                                        copp_meter_get(COPP_ND_NA,
>                                            op->od->nbs->copp,
>                                            meter_groups),
> -                                      &op->nbsp->header_);
> +                                      &op->nbsp->header_,
> +                                      lflow_dep_mgr, op->key, op->od);
>              ds_destroy(&ip6_dst_match);
>              ds_destroy(&nd_target_match);
>          }
> @@ -10088,7 +10239,7 @@ build_lswitch_arp_nd_responder_known_ips(struct ovn_port *op,
>   * (priority 0)*/
>  static void
>  build_lswitch_arp_nd_responder_default(struct ovn_datapath *od,
> -                                       struct hmap *lflows)
> +                                       struct lflow_data *lflows)
>  {
>      ovs_assert(od->nbs);
>      ovn_lflow_add(lflows, od, S_SWITCH_IN_ARP_ND_RSP, 0, "1", "next;");
> @@ -10099,7 +10250,7 @@ build_lswitch_arp_nd_responder_default(struct ovn_datapath *od,
>  static void
>  build_lswitch_arp_nd_service_monitor(const struct ovn_northd_lb *lb,
>                                       const struct hmap *ls_ports,
> -                                     struct hmap *lflows,
> +                                     struct lflow_data *lflows,
>                                       struct ds *actions,
>                                       struct ds *match)
>  {
> @@ -10175,8 +10326,9 @@ build_lswitch_arp_nd_service_monitor(const struct ovn_northd_lb *lb,
>   * priority 100 flows. */
>  static void
>  build_lswitch_dhcp_options_and_response(struct ovn_port *op,
> -                                        struct hmap *lflows,
> -                                        const struct shash *meter_groups)
> +                                        struct lflow_data *lflows,
> +                                        const struct shash *meter_groups,
> +                                        struct objdep_mgr *lflow_dep_mgr)
>  {
>      ovs_assert(op->nbsp);
>      if (!lsp_is_enabled(op->nbsp) || lsp_is_router(op->nbsp)) {
> @@ -10205,19 +10357,19 @@ build_lswitch_dhcp_options_and_response(struct ovn_port *op,
>                  build_dhcpv4_options_flows(
>                      op, &op->lsp_addrs[i],
>                      op->od->localnet_ports[j], is_external,
> -                    meter_groups, lflows);
> +                    meter_groups, lflows, lflow_dep_mgr);
>                  build_dhcpv6_options_flows(
>                      op, &op->lsp_addrs[i],
>                      op->od->localnet_ports[j], is_external,
> -                    meter_groups, lflows);
> +                    meter_groups, lflows, lflow_dep_mgr);
>              }
>          } else {
>              build_dhcpv4_options_flows(op, &op->lsp_addrs[i], op,
>                                         is_external, meter_groups,
> -                                       lflows);
> +                                       lflows, lflow_dep_mgr);
>              build_dhcpv6_options_flows(op, &op->lsp_addrs[i], op,
>                                         is_external, meter_groups,
> -                                       lflows);
> +                                       lflows, lflow_dep_mgr);
>          }
>      }
>  }
> @@ -10230,7 +10382,7 @@ build_lswitch_dhcp_options_and_response(struct ovn_port *op,
>   * (priority 0). */
>  static void
>  build_lswitch_dhcp_and_dns_defaults(struct ovn_datapath *od,
> -                                        struct hmap *lflows)
> +                                        struct lflow_data *lflows)
>  {
>      ovs_assert(od->nbs);
>      ovn_lflow_add(lflows, od, S_SWITCH_IN_DHCP_OPTIONS, 0, "1", "next;");
> @@ -10245,7 +10397,7 @@ build_lswitch_dhcp_and_dns_defaults(struct ovn_datapath *od,
>  */
>  static void
>  build_lswitch_dns_lookup_and_response(struct ovn_datapath *od,
> -                                      struct hmap *lflows,
> +                                      struct lflow_data *lflows,
>                                        const struct shash *meter_groups)
>  {
>      ovs_assert(od->nbs);
> @@ -10276,7 +10428,8 @@ build_lswitch_dns_lookup_and_response(struct ovn_datapath *od,
>   * binding the external ports. */
>  static void
>  build_lswitch_external_port(struct ovn_port *op,
> -                            struct hmap *lflows)
> +                            struct lflow_data *lflows,
> +                            struct objdep_mgr *lflow_dep_mgr)
>  {
>      ovs_assert(op->nbsp);
>      if (!lsp_is_external(op->nbsp)) {
> @@ -10284,7 +10437,7 @@ build_lswitch_external_port(struct ovn_port *op,
>      }
>      for (size_t i = 0; i < op->od->n_localnet_ports; i++) {
>          build_drop_arp_nd_flows_for_unbound_router_ports(
> -            op, op->od->localnet_ports[i], lflows);
> +            op, op->od->localnet_ports[i], lflows, lflow_dep_mgr);
>      }
>  }
>
> @@ -10292,7 +10445,7 @@ build_lswitch_external_port(struct ovn_port *op,
>   * (priority 70 - 100). */
>  static void
>  build_lswitch_destination_lookup_bmcast(struct ovn_datapath *od,
> -                                        struct hmap *lflows,
> +                                        struct lflow_data *lflows,
>                                          struct ds *actions,
>                                          const struct shash *meter_groups)
>  {
> @@ -10383,7 +10536,7 @@ build_lswitch_destination_lookup_bmcast(struct ovn_datapath *od,
>   * (priority 90). */
>  static void
>  build_lswitch_ip_mcast_igmp_mld(struct ovn_igmp_group *igmp_group,
> -                                struct hmap *lflows,
> +                                struct lflow_data *lflows,
>                                  struct ds *actions,
>                                  struct ds *match)
>  {
> @@ -10464,9 +10617,10 @@ build_lswitch_ip_mcast_igmp_mld(struct ovn_igmp_group *igmp_group,
>  /* Ingress table 25: Destination lookup, unicast handling (priority 50), */
>  static void
>  build_lswitch_ip_unicast_lookup(struct ovn_port *op,
> -                                struct hmap *lflows,
> +                                struct lflow_data *lflows,
>                                  struct ds *actions,
> -                                struct ds *match)
> +                                struct ds *match,
> +                                struct objdep_mgr *lflow_dep_mgr)
>  {
>      ovs_assert(op->nbsp);
>      if (lsp_is_external(op->nbsp)) {
> @@ -10479,7 +10633,8 @@ build_lswitch_ip_unicast_lookup(struct ovn_port *op,
>       */
>      if (lsp_is_router(op->nbsp)) {
>          build_lswitch_rport_arp_req_flows(op->peer, op->od, op, lflows,
> -                                          &op->nbsp->header_);
> +                                          &op->nbsp->header_,
> +                                          lflow_dep_mgr);
>      }
>
>      for (size_t i = 0; i < op->nbsp->n_addresses; i++) {
> @@ -10498,10 +10653,12 @@ build_lswitch_ip_unicast_lookup(struct ovn_port *op,
>
>              ds_clear(actions);
>              ds_put_format(actions, action, op->json_key);
> -            ovn_lflow_add_with_hint(lflows, op->od, S_SWITCH_IN_L2_LKUP,
> +            ovn_lflow_add_with_lport_lflow_ref(lflows, op->od,
> +                                    S_SWITCH_IN_L2_LKUP,
>                                      50, ds_cstr(match),
> -                                    ds_cstr(actions),
> -                                    &op->nbsp->header_);
> +                                    ds_cstr(actions), NULL, NULL,
> +                                    &op->nbsp->header_,
> +                                    lflow_dep_mgr, op->key, op->od);
>          } else if (!strcmp(op->nbsp->addresses[i], "unknown")) {
>              continue;
>          } else if (is_dynamic_lsp_address(op->nbsp->addresses[i])) {
> @@ -10516,10 +10673,12 @@ build_lswitch_ip_unicast_lookup(struct ovn_port *op,
>
>              ds_clear(actions);
>              ds_put_format(actions, action, op->json_key);
> -            ovn_lflow_add_with_hint(lflows, op->od, S_SWITCH_IN_L2_LKUP,
> +            ovn_lflow_add_with_lport_lflow_ref(lflows, op->od,
> +                                    S_SWITCH_IN_L2_LKUP,
>                                      50, ds_cstr(match),
> -                                    ds_cstr(actions),
> -                                    &op->nbsp->header_);
> +                                    ds_cstr(actions), NULL, NULL,
> +                                    &op->nbsp->header_,
> +                                    lflow_dep_mgr, op->key, op->od);
>          } else if (!strcmp(op->nbsp->addresses[i], "router")) {
>              if (!op->peer || !op->peer->nbrp
>                  || !ovs_scan(op->peer->nbrp->mac,
> @@ -10571,10 +10730,11 @@ build_lswitch_ip_unicast_lookup(struct ovn_port *op,
>
>              ds_clear(actions);
>              ds_put_format(actions, action, op->json_key);
> -            ovn_lflow_add_with_hint(lflows, op->od,
> +            ovn_lflow_add_with_lport_lflow_ref(lflows, op->od,
>                                      S_SWITCH_IN_L2_LKUP, 50,
>                                      ds_cstr(match), ds_cstr(actions),
> -                                    &op->nbsp->header_);
> +                                    NULL, NULL, &op->nbsp->header_,
> +                                    lflow_dep_mgr, op->key, op->od);
>
>              /* Add ethernet addresses specified in NAT rules on
>               * distributed logical routers. */
> @@ -10594,11 +10754,13 @@ build_lswitch_ip_unicast_lookup(struct ovn_port *op,
>
>                          ds_clear(actions);
>                          ds_put_format(actions, action, op->json_key);
> -                        ovn_lflow_add_with_hint(lflows, op->od,
> +                        ovn_lflow_add_with_lport_lflow_ref(lflows, op->od,
>                                                  S_SWITCH_IN_L2_LKUP, 50,
>                                                  ds_cstr(match),
>                                                  ds_cstr(actions),
> -                                                &op->nbsp->header_);
> +                                                NULL, NULL, &op->nbsp->header_,
> +                                                lflow_dep_mgr, op->key,
> +                                                op->od);
>                      }
>                  }
>              }
> @@ -10847,7 +11009,7 @@ get_outport_for_routing_policy_nexthop(struct ovn_datapath *od,
>  }
>
>  static void
> -build_routing_policy_flow(struct hmap *lflows, struct ovn_datapath *od,
> +build_routing_policy_flow(struct lflow_data *lflows, struct ovn_datapath *od,
>                            const struct hmap *lr_ports,
>                            const struct nbrec_logical_router_policy *rule,
>                            const struct ovsdb_idl_row *stage_hint)
> @@ -10912,7 +11074,8 @@ build_routing_policy_flow(struct hmap *lflows, struct ovn_datapath *od,
>  }
>
>  static void
> -build_ecmp_routing_policy_flows(struct hmap *lflows, struct ovn_datapath *od,
> +build_ecmp_routing_policy_flows(struct lflow_data *lflows,
> +                                struct ovn_datapath *od,
>                                  const struct hmap *lr_ports,
>                                  const struct nbrec_logical_router_policy *rule,
>                                  uint16_t ecmp_group_id)
> @@ -11048,7 +11211,7 @@ get_route_table_id(struct simap *route_tables, const char *route_table_name)
>  }
>
>  static void
> -build_route_table_lflow(struct ovn_datapath *od, struct hmap *lflows,
> +build_route_table_lflow(struct ovn_datapath *od, struct lflow_data *lflows,
>                          struct nbrec_logical_router_port *lrp,
>                          struct simap *route_tables)
>  {
> @@ -11459,7 +11622,7 @@ find_static_route_outport(struct ovn_datapath *od, const struct hmap *lr_ports,
>  }
>
>  static void
> -add_ecmp_symmetric_reply_flows(struct hmap *lflows,
> +add_ecmp_symmetric_reply_flows(struct lflow_data *lflows,
>                                 struct ovn_datapath *od,
>                                 bool ct_masked_mark,
>                                 const char *port_ip,
> @@ -11632,7 +11795,7 @@ add_ecmp_symmetric_reply_flows(struct hmap *lflows,
>  }
>
>  static void
> -build_ecmp_route_flow(struct hmap *lflows, struct ovn_datapath *od,
> +build_ecmp_route_flow(struct lflow_data *lflows, struct ovn_datapath *od,
>                        bool ct_masked_mark, const struct hmap *lr_ports,
>                        struct ecmp_groups_node *eg)
>
> @@ -11719,7 +11882,7 @@ build_ecmp_route_flow(struct hmap *lflows, struct ovn_datapath *od,
>  }
>
>  static void
> -add_route(struct hmap *lflows, struct ovn_datapath *od,
> +add_route(struct lflow_data *lflows, struct ovn_datapath *od,
>            const struct ovn_port *op, const char *lrp_addr_s,
>            const char *network_s, int plen, const char *gateway,
>            bool is_src_route, const uint32_t rtb_id,
> @@ -11782,7 +11945,7 @@ add_route(struct hmap *lflows, struct ovn_datapath *od,
>  }
>
>  static void
> -build_static_route_flow(struct hmap *lflows, struct ovn_datapath *od,
> +build_static_route_flow(struct lflow_data *lflows, struct ovn_datapath *od,
>                          const struct hmap *lr_ports,
>                          const struct parsed_route *route_)
>  {
> @@ -11892,7 +12055,7 @@ struct lrouter_nat_lb_flows_ctx {
>
>      int prio;
>
> -    struct hmap *lflows;
> +    struct lflow_data *lflows;
>      const struct shash *meter_groups;
>  };
>
> @@ -12022,7 +12185,7 @@ build_lrouter_nat_flows_for_lb(struct ovn_lb_vip *lb_vip,
>                                 struct ovn_lb_datapaths *lb_dps,
>                                 struct ovn_northd_lb_vip *vips_nb,
>                                 const struct ovn_datapaths *lr_datapaths,
> -                               struct hmap *lflows,
> +                               struct lflow_data *lflows,
>                                 struct ds *match, struct ds *action,
>                                 const struct shash *meter_groups,
>                                 const struct chassis_features *features,
> @@ -12202,7 +12365,7 @@ build_lrouter_nat_flows_for_lb(struct ovn_lb_vip *lb_vip,
>
>  static void
>  build_lswitch_flows_for_lb(struct ovn_lb_datapaths *lb_dps,
> -                           struct hmap *lflows,
> +                           struct lflow_data *lflows,
>                             const struct shash *meter_groups,
>                             const struct ovn_datapaths *ls_datapaths,
>                             const struct chassis_features *features,
> @@ -12263,7 +12426,7 @@ build_lswitch_flows_for_lb(struct ovn_lb_datapaths *lb_dps,
>   */
>  static void
>  build_lrouter_defrag_flows_for_lb(struct ovn_lb_datapaths *lb_dps,
> -                                  struct hmap *lflows,
> +                                  struct lflow_data *lflows,
>                                    const struct ovn_datapaths *lr_datapaths,
>                                    struct ds *match)
>  {
> @@ -12289,7 +12452,7 @@ build_lrouter_defrag_flows_for_lb(struct ovn_lb_datapaths *lb_dps,
>
>  static void
>  build_lrouter_flows_for_lb(struct ovn_lb_datapaths *lb_dps,
> -                           struct hmap *lflows,
> +                           struct lflow_data *lflows,
>                             const struct shash *meter_groups,
>                             const struct ovn_datapaths *lr_datapaths,
>                             const struct chassis_features *features,
> @@ -12446,7 +12609,7 @@ lrouter_dnat_and_snat_is_stateless(const struct nbrec_nat *nat)
>   */
>  static inline void
>  lrouter_nat_add_ext_ip_match(struct ovn_datapath *od,
> -                             struct hmap *lflows, struct ds *match,
> +                             struct lflow_data *lflows, struct ds *match,
>                               const struct nbrec_nat *nat,
>                               bool is_v6, bool is_src, int cidr_bits)
>  {
> @@ -12513,7 +12676,7 @@ build_lrouter_arp_flow(struct ovn_datapath *od, struct ovn_port *op,
>                         const char *ip_address, const char *eth_addr,
>                         struct ds *extra_match, bool drop, uint16_t priority,
>                         const struct ovsdb_idl_row *hint,
> -                       struct hmap *lflows)
> +                       struct lflow_data *lflows)
>  {
>      struct ds match = DS_EMPTY_INITIALIZER;
>      struct ds actions = DS_EMPTY_INITIALIZER;
> @@ -12563,7 +12726,8 @@ build_lrouter_nd_flow(struct ovn_datapath *od, struct ovn_port *op,
>                        const char *sn_ip_address, const char *eth_addr,
>                        struct ds *extra_match, bool drop, uint16_t priority,
>                        const struct ovsdb_idl_row *hint,
> -                      struct hmap *lflows, const struct shash *meter_groups)
> +                      struct lflow_data *lflows,
> +                      const struct shash *meter_groups)
>  {
>      struct ds match = DS_EMPTY_INITIALIZER;
>      struct ds actions = DS_EMPTY_INITIALIZER;
> @@ -12614,7 +12778,7 @@ build_lrouter_nd_flow(struct ovn_datapath *od, struct ovn_port *op,
>  static void
>  build_lrouter_nat_arp_nd_flow(struct ovn_datapath *od,
>                                struct ovn_nat *nat_entry,
> -                              struct hmap *lflows,
> +                              struct lflow_data *lflows,
>                                const struct shash *meter_groups)
>  {
>      struct lport_addresses *ext_addrs = &nat_entry->ext_addrs;
> @@ -12637,7 +12801,7 @@ build_lrouter_nat_arp_nd_flow(struct ovn_datapath *od,
>  static void
>  build_lrouter_port_nat_arp_nd_flow(struct ovn_port *op,
>                                     struct ovn_nat *nat_entry,
> -                                   struct hmap *lflows,
> +                                   struct lflow_data *lflows,
>                                     const struct shash *meter_groups)
>  {
>      struct lport_addresses *ext_addrs = &nat_entry->ext_addrs;
> @@ -12709,7 +12873,7 @@ build_lrouter_port_nat_arp_nd_flow(struct ovn_port *op,
>  static void
>  build_lrouter_drop_own_dest(struct ovn_port *op, enum ovn_stage stage,
>                              uint16_t priority, bool drop_snat_ip,
> -                            struct hmap *lflows)
> +                            struct lflow_data *lflows)
>  {
>      struct ds match_ips = DS_EMPTY_INITIALIZER;
>
> @@ -12772,7 +12936,8 @@ build_lrouter_drop_own_dest(struct ovn_port *op, enum ovn_stage stage,
>  }
>
>  static void
> -build_lrouter_force_snat_flows(struct hmap *lflows, struct ovn_datapath *od,
> +build_lrouter_force_snat_flows(struct lflow_data *lflows,
> +                               struct ovn_datapath *od,
>                                 const char *ip_version, const char *ip_addr,
>                                 const char *context)
>  {
> @@ -12799,7 +12964,7 @@ build_lrouter_force_snat_flows(struct hmap *lflows, struct ovn_datapath *od,
>
>  static void
>  build_lrouter_force_snat_flows_op(struct ovn_port *op,
> -                                  struct hmap *lflows,
> +                                  struct lflow_data *lflows,
>                                    struct ds *match, struct ds *actions)
>  {
>      ovs_assert(op->nbrp);
> @@ -12871,7 +13036,7 @@ build_lrouter_force_snat_flows_op(struct ovn_port *op,
>  }
>
>  static void
> -build_lrouter_bfd_flows(struct hmap *lflows, struct ovn_port *op,
> +build_lrouter_bfd_flows(struct lflow_data *lflows, struct ovn_port *op,
>                          const struct shash *meter_groups)
>  {
>      if (!op->has_bfd) {
> @@ -12926,7 +13091,7 @@ build_lrouter_bfd_flows(struct hmap *lflows, struct ovn_port *op,
>   */
>  static void
>  build_adm_ctrl_flows_for_lrouter(
> -        struct ovn_datapath *od, struct hmap *lflows)
> +        struct ovn_datapath *od, struct lflow_data *lflows)
>  {
>      ovs_assert(od->nbr);
>      /* Logical VLANs not supported.
> @@ -12964,7 +13129,7 @@ build_gateway_get_l2_hdr_size(struct ovn_port *op)
>   * function.
>   */
>  static void OVS_PRINTF_FORMAT(9, 10)
> -build_gateway_mtu_flow(struct hmap *lflows, struct ovn_port *op,
> +build_gateway_mtu_flow(struct lflow_data *lflows, struct ovn_port *op,
>                         enum ovn_stage stage, uint16_t prio_low,
>                         uint16_t prio_high, struct ds *match,
>                         struct ds *actions, const struct ovsdb_idl_row *hint,
> @@ -13025,7 +13190,7 @@ consider_l3dgw_port_is_centralized(struct ovn_port *op)
>   */
>  static void
>  build_adm_ctrl_flows_for_lrouter_port(
> -        struct ovn_port *op, struct hmap *lflows,
> +        struct ovn_port *op, struct lflow_data *lflows,
>          struct ds *match, struct ds *actions)
>  {
>      ovs_assert(op->nbrp);
> @@ -13079,7 +13244,7 @@ build_adm_ctrl_flows_for_lrouter_port(
>   * lflows for logical routers. */
>  static void
>  build_neigh_learning_flows_for_lrouter(
> -        struct ovn_datapath *od, struct hmap *lflows,
> +        struct ovn_datapath *od, struct lflow_data *lflows,
>          struct ds *match, struct ds *actions,
>          const struct shash *meter_groups)
>  {
> @@ -13210,7 +13375,7 @@ build_neigh_learning_flows_for_lrouter(
>   * for logical router ports. */
>  static void
>  build_neigh_learning_flows_for_lrouter_port(
> -        struct ovn_port *op, struct hmap *lflows,
> +        struct ovn_port *op, struct lflow_data *lflows,
>          struct ds *match, struct ds *actions)
>  {
>      ovs_assert(op->nbrp);
> @@ -13272,7 +13437,7 @@ build_neigh_learning_flows_for_lrouter_port(
>   * Adv (RA) options and response. */
>  static void
>  build_ND_RA_flows_for_lrouter_port(
> -        struct ovn_port *op, struct hmap *lflows,
> +        struct ovn_port *op, struct lflow_data *lflows,
>          struct ds *match, struct ds *actions,
>          const struct shash *meter_groups)
>  {
> @@ -13387,7 +13552,8 @@ build_ND_RA_flows_for_lrouter_port(
>  /* Logical router ingress table ND_RA_OPTIONS & ND_RA_RESPONSE: RS
>   * responder, by default goto next. (priority 0). */
>  static void
> -build_ND_RA_flows_for_lrouter(struct ovn_datapath *od, struct hmap *lflows)
> +build_ND_RA_flows_for_lrouter(struct ovn_datapath *od,
> +                              struct lflow_data *lflows)
>  {
>      ovs_assert(od->nbr);
>      ovn_lflow_add(lflows, od, S_ROUTER_IN_ND_RA_OPTIONS, 0, "1", "next;");
> @@ -13398,7 +13564,7 @@ build_ND_RA_flows_for_lrouter(struct ovn_datapath *od, struct hmap *lflows)
>   * by default goto next. (priority 0). */
>  static void
>  build_ip_routing_pre_flows_for_lrouter(struct ovn_datapath *od,
> -                                       struct hmap *lflows)
> +                                       struct lflow_data *lflows)
>  {
>      ovs_assert(od->nbr);
>      ovn_lflow_add(lflows, od, S_ROUTER_IN_IP_ROUTING_PRE, 0, "1",
> @@ -13426,7 +13592,7 @@ build_ip_routing_pre_flows_for_lrouter(struct ovn_datapath *od,
>   */
>  static void
>  build_ip_routing_flows_for_lrp(
> -        struct ovn_port *op, struct hmap *lflows)
> +        struct ovn_port *op, struct lflow_data *lflows)
>  {
>      ovs_assert(op->nbrp);
>      for (int i = 0; i < op->lrp_networks.n_ipv4_addrs; i++) {
> @@ -13452,8 +13618,9 @@ build_ip_routing_flows_for_lrp(
>   * ports, add routes to the LSP's peer router.
>   */
>  static void
> -build_ip_routing_flows_for_router_type_lsp(
> -        struct ovn_port *op, const struct hmap *lr_ports, struct hmap *lflows)
> +build_ip_routing_flows_for_router_type_lsp(struct ovn_port *op,
> +                                           const struct hmap *lr_ports,
> +                                           struct lflow_data *lflows)
>  {
>      ovs_assert(op->nbsp);
>      if (!lsp_is_router(op->nbsp)) {
> @@ -13490,7 +13657,7 @@ build_ip_routing_flows_for_router_type_lsp(
>  static void
>  build_static_route_flows_for_lrouter(
>          struct ovn_datapath *od, const struct chassis_features *features,
> -        struct hmap *lflows, const struct hmap *lr_ports,
> +        struct lflow_data *lflows, const struct hmap *lr_ports,
>          const struct hmap *bfd_connections)
>  {
>      ovs_assert(od->nbr);
> @@ -13554,7 +13721,7 @@ build_static_route_flows_for_lrouter(
>   */
>  static void
>  build_mcast_lookup_flows_for_lrouter(
> -        struct ovn_datapath *od, struct hmap *lflows,
> +        struct ovn_datapath *od, struct lflow_data *lflows,
>          struct ds *match, struct ds *actions)
>  {
>      ovs_assert(od->nbr);
> @@ -13655,7 +13822,7 @@ build_mcast_lookup_flows_for_lrouter(
>   * advances to the next table for ARP/ND resolution. */
>  static void
>  build_ingress_policy_flows_for_lrouter(
> -        struct ovn_datapath *od, struct hmap *lflows,
> +        struct ovn_datapath *od, struct lflow_data *lflows,
>          const struct hmap *lr_ports)
>  {
>      ovs_assert(od->nbr);
> @@ -13689,7 +13856,7 @@ build_ingress_policy_flows_for_lrouter(
>  /* Local router ingress table ARP_RESOLVE: ARP Resolution. */
>  static void
>  build_arp_resolve_flows_for_lrouter(
> -        struct ovn_datapath *od, struct hmap *lflows)
> +        struct ovn_datapath *od, struct lflow_data *lflows)
>  {
>      ovs_assert(od->nbr);
>      /* Multicast packets already have the outport set so just advance to
> @@ -13707,7 +13874,8 @@ build_arp_resolve_flows_for_lrouter(
>  }
>
>  static void
> -routable_addresses_to_lflows(struct hmap *lflows, struct ovn_port *router_port,
> +routable_addresses_to_lflows(struct lflow_data *lflows,
> +                             struct ovn_port *router_port,
>                               struct ovn_port *peer, struct ds *match,
>                               struct ds *actions)
>  {
> @@ -13750,7 +13918,7 @@ routable_addresses_to_lflows(struct hmap *lflows, struct ovn_port *router_port,
>  /* This function adds ARP resolve flows related to a LRP. */
>  static void
>  build_arp_resolve_flows_for_lrp(
> -        struct ovn_port *op, struct hmap *lflows,
> +        struct ovn_port *op, struct lflow_data *lflows,
>          struct ds *match, struct ds *actions)
>  {
>      ovs_assert(op->nbrp);
> @@ -13834,9 +14002,10 @@ build_arp_resolve_flows_for_lrp(
>  /* This function adds ARP resolve flows related to a LSP. */
>  static void
>  build_arp_resolve_flows_for_lsp(
> -        struct ovn_port *op, struct hmap *lflows,
> +        struct ovn_port *op, struct lflow_data *lflows,
>          const struct hmap *lr_ports,
> -        struct ds *match, struct ds *actions)
> +        struct ds *match, struct ds *actions,
> +        struct objdep_mgr *lflow_dep_mgr)
>  {
>      ovs_assert(op->nbsp);
>      if (!lsp_is_enabled(op->nbsp)) {
> @@ -13876,11 +14045,13 @@ build_arp_resolve_flows_for_lsp(
>
>                      ds_clear(actions);
>                      ds_put_format(actions, "eth.dst = %s; next;", ea_s);
> -                    ovn_lflow_add_with_hint(lflows, peer->od,
> +                    ovn_lflow_add_with_lport_lflow_ref(lflows, peer->od,
>                                              S_ROUTER_IN_ARP_RESOLVE, 100,
>                                              ds_cstr(match),
>                                              ds_cstr(actions),
> -                                            &op->nbsp->header_);
> +                                            NULL, NULL,
> +                                            &op->nbsp->header_,
> +                                            lflow_dep_mgr, op->key, op->od);
>                  }
>              }
>
> @@ -13907,11 +14078,13 @@ build_arp_resolve_flows_for_lsp(
>
>                      ds_clear(actions);
>                      ds_put_format(actions, "eth.dst = %s; next;", ea_s);
> -                    ovn_lflow_add_with_hint(lflows, peer->od,
> +                    ovn_lflow_add_with_lport_lflow_ref(lflows, peer->od,
>                                              S_ROUTER_IN_ARP_RESOLVE, 100,
>                                              ds_cstr(match),
>                                              ds_cstr(actions),
> -                                            &op->nbsp->header_);
> +                                            NULL, NULL,
> +                                            &op->nbsp->header_,
> +                                            lflow_dep_mgr, op->key, op->od);
>                  }
>              }
>          }
> @@ -13987,7 +14160,8 @@ build_arp_resolve_flows_for_lsp(
>  }
>
>  static void
> -build_icmperr_pkt_big_flows(struct ovn_port *op, int mtu, struct hmap *lflows,
> +build_icmperr_pkt_big_flows(struct ovn_port *op, int mtu,
> +                            struct lflow_data *lflows,
>                              const struct shash *meter_groups, struct ds *match,
>                              struct ds *actions, enum ovn_stage stage,
>                              struct ovn_port *outport)
> @@ -14066,7 +14240,7 @@ build_icmperr_pkt_big_flows(struct ovn_port *op, int mtu, struct hmap *lflows,
>
>  static void
>  build_check_pkt_len_flows_for_lrp(struct ovn_port *op,
> -                                  struct hmap *lflows,
> +                                  struct lflow_data *lflows,
>                                    const struct hmap *lr_ports,
>                                    const struct shash *meter_groups,
>                                    struct ds *match,
> @@ -14116,7 +14290,7 @@ build_check_pkt_len_flows_for_lrp(struct ovn_port *op,
>   * */
>  static void
>  build_check_pkt_len_flows_for_lrouter(
> -        struct ovn_datapath *od, struct hmap *lflows,
> +        struct ovn_datapath *od, struct lflow_data *lflows,
>          const struct hmap *lr_ports,
>          struct ds *match, struct ds *actions,
>          const struct shash *meter_groups)
> @@ -14143,7 +14317,7 @@ build_check_pkt_len_flows_for_lrouter(
>  /* Logical router ingress table GW_REDIRECT: Gateway redirect. */
>  static void
>  build_gateway_redirect_flows_for_lrouter(
> -        struct ovn_datapath *od, struct hmap *lflows,
> +        struct ovn_datapath *od, struct lflow_data *lflows,
>          struct ds *match, struct ds *actions)
>  {
>      ovs_assert(od->nbr);
> @@ -14228,7 +14402,7 @@ build_gateway_redirect_flows_for_lrouter(
>   * and sends an ARP/IPv6 NA request (priority 100). */
>  static void
>  build_arp_request_flows_for_lrouter(
> -        struct ovn_datapath *od, struct hmap *lflows,
> +        struct ovn_datapath *od, struct lflow_data *lflows,
>          struct ds *match, struct ds *actions,
>          const struct shash *meter_groups)
>  {
> @@ -14306,7 +14480,7 @@ build_arp_request_flows_for_lrouter(
>   */
>  static void
>  build_egress_delivery_flows_for_lrouter_port(
> -        struct ovn_port *op, struct hmap *lflows,
> +        struct ovn_port *op, struct lflow_data *lflows,
>          struct ds *match, struct ds *actions)
>  {
>      ovs_assert(op->nbrp);
> @@ -14348,7 +14522,7 @@ build_egress_delivery_flows_for_lrouter_port(
>
>  static void
>  build_misc_local_traffic_drop_flows_for_lrouter(
> -        struct ovn_datapath *od, struct hmap *lflows)
> +        struct ovn_datapath *od, struct lflow_data *lflows)
>  {
>      ovs_assert(od->nbr);
>      /* Allow IGMP and MLD packets (with TTL = 1) if the router is
> @@ -14430,7 +14604,7 @@ build_misc_local_traffic_drop_flows_for_lrouter(
>
>  static void
>  build_dhcpv6_reply_flows_for_lrouter_port(
> -        struct ovn_port *op, struct hmap *lflows,
> +        struct ovn_port *op, struct lflow_data *lflows,
>          struct ds *match)
>  {
>      ovs_assert(op->nbrp);
> @@ -14450,7 +14624,7 @@ build_dhcpv6_reply_flows_for_lrouter_port(
>
>  static void
>  build_ipv6_input_flows_for_lrouter_port(
> -        struct ovn_port *op, struct hmap *lflows,
> +        struct ovn_port *op, struct lflow_data *lflows,
>          struct ds *match, struct ds *actions,
>          const struct shash *meter_groups)
>  {
> @@ -14618,7 +14792,7 @@ build_ipv6_input_flows_for_lrouter_port(
>
>  static void
>  build_lrouter_arp_nd_for_datapath(struct ovn_datapath *od,
> -                                  struct hmap *lflows,
> +                                  struct lflow_data *lflows,
>                                    const struct shash *meter_groups)
>  {
>      ovs_assert(od->nbr);
> @@ -14666,7 +14840,7 @@ build_lrouter_arp_nd_for_datapath(struct ovn_datapath *od,
>  /* Logical router ingress table 3: IP Input for IPv4. */
>  static void
>  build_lrouter_ipv4_ip_input(struct ovn_port *op,
> -                            struct hmap *lflows,
> +                            struct lflow_data *lflows,
>                              struct ds *match, struct ds *actions,
>                              const struct shash *meter_groups)
>  {
> @@ -14987,7 +15161,7 @@ build_lrouter_in_unsnat_match(struct ovn_datapath *od,
>  }
>
>  static void
> -build_lrouter_in_unsnat_stateless_flow(struct hmap *lflows,
> +build_lrouter_in_unsnat_stateless_flow(struct lflow_data *lflows,
>                                         struct ovn_datapath *od,
>                                         const struct nbrec_nat *nat,
>                                         struct ds *match,
> @@ -15009,7 +15183,7 @@ build_lrouter_in_unsnat_stateless_flow(struct hmap *lflows,
>  }
>
>  static void
> -build_lrouter_in_unsnat_in_czone_flow(struct hmap *lflows,
> +build_lrouter_in_unsnat_in_czone_flow(struct lflow_data *lflows,
>                                        struct ovn_datapath *od,
>                                        const struct nbrec_nat *nat,
>                                        struct ds *match, bool distributed_nat,
> @@ -15043,7 +15217,8 @@ build_lrouter_in_unsnat_in_czone_flow(struct hmap *lflows,
>  }
>
>  static void
> -build_lrouter_in_unsnat_flow(struct hmap *lflows, struct ovn_datapath *od,
> +build_lrouter_in_unsnat_flow(struct lflow_data *lflows,
> +                             struct ovn_datapath *od,
>                               const struct nbrec_nat *nat, struct ds *match,
>                               bool distributed_nat, bool is_v6,
>                               struct ovn_port *l3dgw_port)
> @@ -15064,7 +15239,7 @@ build_lrouter_in_unsnat_flow(struct hmap *lflows, struct ovn_datapath *od,
>  }
>
>  static void
> -build_lrouter_in_dnat_flow(struct hmap *lflows, struct ovn_datapath *od,
> +build_lrouter_in_dnat_flow(struct lflow_data *lflows, struct ovn_datapath *od,
>                             const struct nbrec_nat *nat, struct ds *match,
>                             struct ds *actions, bool distributed_nat,
>                             int cidr_bits, bool is_v6,
> @@ -15134,7 +15309,8 @@ build_lrouter_in_dnat_flow(struct hmap *lflows, struct ovn_datapath *od,
>  }
>
>  static void
> -build_lrouter_out_undnat_flow(struct hmap *lflows, struct ovn_datapath *od,
> +build_lrouter_out_undnat_flow(struct lflow_data *lflows,
> +                              struct ovn_datapath *od,
>                                const struct nbrec_nat *nat, struct ds *match,
>                                struct ds *actions, bool distributed_nat,
>                                struct eth_addr mac, bool is_v6,
> @@ -15184,7 +15360,8 @@ build_lrouter_out_undnat_flow(struct hmap *lflows, struct ovn_datapath *od,
>  }
>
>  static void
> -build_lrouter_out_is_dnat_local(struct hmap *lflows, struct ovn_datapath *od,
> +build_lrouter_out_is_dnat_local(struct lflow_data *lflows,
> +                                struct ovn_datapath *od,
>                                  const struct nbrec_nat *nat, struct ds *match,
>                                  struct ds *actions, bool distributed_nat,
>                                  bool is_v6, struct ovn_port *l3dgw_port)
> @@ -15214,7 +15391,8 @@ build_lrouter_out_is_dnat_local(struct hmap *lflows, struct ovn_datapath *od,
>  }
>
>  static void
> -build_lrouter_out_snat_match(struct hmap *lflows, struct ovn_datapath *od,
> +build_lrouter_out_snat_match(struct lflow_data *lflows,
> +                             struct ovn_datapath *od,
>                               const struct nbrec_nat *nat, struct ds *match,
>                               bool distributed_nat, int cidr_bits, bool is_v6,
>                               struct ovn_port *l3dgw_port)
> @@ -15242,7 +15420,7 @@ build_lrouter_out_snat_match(struct hmap *lflows, struct ovn_datapath *od,
>  }
>
>  static void
> -build_lrouter_out_snat_stateless_flow(struct hmap *lflows,
> +build_lrouter_out_snat_stateless_flow(struct lflow_data *lflows,
>                                        struct ovn_datapath *od,
>                                        const struct nbrec_nat *nat,
>                                        struct ds *match, struct ds *actions,
> @@ -15285,7 +15463,7 @@ build_lrouter_out_snat_stateless_flow(struct hmap *lflows,
>  }
>
>  static void
> -build_lrouter_out_snat_in_czone_flow(struct hmap *lflows,
> +build_lrouter_out_snat_in_czone_flow(struct lflow_data *lflows,
>                                       struct ovn_datapath *od,
>                                       const struct nbrec_nat *nat,
>                                       struct ds *match,
> @@ -15346,7 +15524,7 @@ build_lrouter_out_snat_in_czone_flow(struct hmap *lflows,
>  }
>
>  static void
> -build_lrouter_out_snat_flow(struct hmap *lflows, struct ovn_datapath *od,
> +build_lrouter_out_snat_flow(struct lflow_data *lflows, struct ovn_datapath *od,
>                              const struct nbrec_nat *nat, struct ds *match,
>                              struct ds *actions, bool distributed_nat,
>                              struct eth_addr mac, int cidr_bits, bool is_v6,
> @@ -15393,7 +15571,7 @@ build_lrouter_out_snat_flow(struct hmap *lflows, struct ovn_datapath *od,
>  }
>
>  static void
> -build_lrouter_ingress_nat_check_pkt_len(struct hmap *lflows,
> +build_lrouter_ingress_nat_check_pkt_len(struct lflow_data *lflows,
>                                          const struct nbrec_nat *nat,
>                                          struct ovn_datapath *od, bool is_v6,
>                                          struct ds *match, struct ds *actions,
> @@ -15464,7 +15642,7 @@ build_lrouter_ingress_nat_check_pkt_len(struct hmap *lflows,
>  }
>
>  static void
> -build_lrouter_ingress_flow(struct hmap *lflows, struct ovn_datapath *od,
> +build_lrouter_ingress_flow(struct lflow_data *lflows, struct ovn_datapath *od,
>                             const struct nbrec_nat *nat, struct ds *match,
>                             struct ds *actions, struct eth_addr mac,
>                             bool distributed_nat, bool is_v6,
> @@ -15642,7 +15820,8 @@ lrouter_check_nat_entry(struct ovn_datapath *od, const struct nbrec_nat *nat,
>
>  /* NAT, Defrag and load balancing. */
>  static void
> -build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, struct hmap *lflows,
> +build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od,
> +                                struct lflow_data *lflows,
>                                  const struct hmap *ls_ports,
>                                  const struct hmap *lr_ports,
>                                  struct ds *match,
> @@ -16043,7 +16222,7 @@ struct lswitch_flow_build_info {
>      const struct hmap *ls_ports;
>      const struct hmap *lr_ports;
>      const struct hmap *port_groups;
> -    struct hmap *lflows;
> +    struct lflow_data *lflows;
>      struct hmap *igmp_groups;
>      const struct shash *meter_groups;
>      const struct hmap *lb_dps_map;
> @@ -16131,27 +16310,28 @@ build_lswitch_and_lrouter_iterate_by_lsp(struct ovn_port *op,
>                                           const struct shash *meter_groups,
>                                           struct ds *match,
>                                           struct ds *actions,
> -                                         struct hmap *lflows)
> +                                         struct lflow_data *lflows)
>  {
>      ovs_assert(op->nbsp);
> -    start_collecting_lflows();
>
>      /* Build Logical Switch Flows. */
> -    build_lswitch_port_sec_op(op, lflows, actions, match);
> -    build_lswitch_learn_fdb_op(op, lflows, actions, match);
> -    build_lswitch_arp_nd_responder_skip_local(op, lflows, match);
> +    build_lswitch_port_sec_op(op, lflows, actions, match, &op->lflow_dep_mgr);
> +    build_lswitch_learn_fdb_op(op, lflows, actions, match, &op->lflow_dep_mgr);
> +    build_lswitch_arp_nd_responder_skip_local(op, lflows, match,
> +                                              &op->lflow_dep_mgr);
>      build_lswitch_arp_nd_responder_known_ips(op, lflows, ls_ports,
> -                                             meter_groups, actions, match);
> -    build_lswitch_dhcp_options_and_response(op, lflows, meter_groups);
> -    build_lswitch_external_port(op, lflows);
> -    build_lswitch_ip_unicast_lookup(op, lflows, actions, match);
> +                                             meter_groups, actions, match,
> +                                             &op->lflow_dep_mgr);
> +    build_lswitch_dhcp_options_and_response(op, lflows, meter_groups,
> +                                            &op->lflow_dep_mgr);
> +    build_lswitch_external_port(op, lflows, &op->lflow_dep_mgr);
> +    build_lswitch_ip_unicast_lookup(op, lflows, actions, match,
> +                                    &op->lflow_dep_mgr);
>
>      /* Build Logical Router Flows. */
>      build_ip_routing_flows_for_router_type_lsp(op, lr_ports, lflows);
> -    build_arp_resolve_flows_for_lsp(op, lflows, lr_ports, match, actions);
> -
> -    link_ovn_port_to_lflows(op, &collected_lflows);
> -    end_collecting_lflows();
> +    build_arp_resolve_flows_for_lsp(op, lflows, lr_ports, match, actions,
> +                                    &op->lflow_dep_mgr);
>  }
>
>  /* Helper function to combine all lflow generation which is iterated by logical
> @@ -16347,7 +16527,7 @@ build_lswitch_and_lrouter_flows(const struct ovn_datapaths *ls_datapaths,
>                                  const struct hmap *ls_ports,
>                                  const struct hmap *lr_ports,
>                                  const struct hmap *port_groups,
> -                                struct hmap *lflows,
> +                                struct lflow_data *lflows,
>                                  struct hmap *igmp_groups,
>                                  const struct shash *meter_groups,
>                                  const struct hmap *lb_dps_map,
> @@ -16392,7 +16572,10 @@ build_lswitch_and_lrouter_flows(const struct ovn_datapaths *ls_datapaths,
>
>          /* Run thread pool. */
>          run_pool_callback(build_lflows_pool, NULL, NULL, noop_callback);
> -        fix_flow_map_size(lflows, lsiv, build_lflows_pool->size);
> +        fix_flow_map_size(&lflows->lflows_match_map, lsiv,
> +                          build_lflows_pool->size);
> +        fix_flow_map_size(&lflows->lflows_hash_map, lsiv,
> +                          build_lflows_pool->size);
>
>          for (index = 0; index < build_lflows_pool->size; index++) {
>              ds_destroy(&lsiv[index].match);
> @@ -16483,17 +16666,33 @@ static ssize_t max_seen_lflow_size = 128;
>  void
>  lflow_data_init(struct lflow_data *data)
>  {
> -    fast_hmap_size_for(&data->lflows, max_seen_lflow_size);
> +    fast_hmap_size_for(&data->lflows_match_map, max_seen_lflow_size);
> +    fast_hmap_size_for(&data->lflows_hash_map, max_seen_lflow_size);
> +    hmap_init(&data->ls_dp_groups);
> +    hmap_init(&data->lr_dp_groups);
>  }
>
>  void
>  lflow_data_destroy(struct lflow_data *data)
>  {
>      struct ovn_lflow *lflow;
> -    HMAP_FOR_EACH_SAFE (lflow, hmap_node, &data->lflows) {
> -        ovn_lflow_destroy(&data->lflows, lflow);
> +    HMAP_FOR_EACH_SAFE (lflow, hmap_node, &data->lflows_match_map) {
> +        ovn_lflow_destroy(data, lflow);
> +    }
> +    hmap_destroy(&data->lflows_match_map);
> +    hmap_destroy(&data->lflows_hash_map);
> +
> +    struct ovn_dp_group *dpg;
> +    HMAP_FOR_EACH_POP (dpg, node, &data->ls_dp_groups) {
> +        bitmap_free(dpg->bitmap);
> +        free(dpg);
>      }
> -    hmap_destroy(&data->lflows);
> +    hmap_destroy(&data->ls_dp_groups);
> +    HMAP_FOR_EACH_POP (dpg, node, &data->lr_dp_groups) {
> +        bitmap_free(dpg->bitmap);
> +        free(dpg);
> +    }
> +    hmap_destroy(&data->lr_dp_groups);
>  }
>
>  void run_update_worker_pool(int n_threads)
> @@ -16537,11 +16736,172 @@ create_sb_multicast_group(struct ovsdb_idl_txn *ovnsb_txn,
>      return sbmc;
>  }
>
> +static void
> +sync_lflow_to_sb(struct ovsdb_idl_txn *ovnsb_txn,
> +                 struct lflow_input *lflow_input,
> +                 struct lflow_data *lflow_data,
> +                 struct ovn_lflow *lflow,
> +                 const struct sbrec_logical_flow *sbflow)
> +{
> +    size_t n_datapaths;
> +    struct ovn_datapath **datapaths_array;
> +    struct hmap *dp_groups;
> +    bool is_switch;
> +    if (ovn_stage_to_datapath_type(lflow->stage) == DP_SWITCH) {
> +        n_datapaths = ods_size(lflow_input->ls_datapaths);
> +        datapaths_array = lflow_input->ls_datapaths->array;
> +        dp_groups = &lflow_data->ls_dp_groups;
> +        is_switch = true;
> +    } else {
> +        n_datapaths = ods_size(lflow_input->lr_datapaths);
> +        datapaths_array = lflow_input->lr_datapaths->array;
> +        dp_groups = &lflow_data->lr_dp_groups;
> +        is_switch = false;
> +    }
> +
> +    lflow->n_ods = bitmap_count1(lflow->dpg_bitmap, n_datapaths);
> +    ovs_assert(lflow->n_ods);
> +
> +    struct ovn_dp_group *pre_sync_dpg = lflow->dpg;
> +    if (lflow->n_ods == 1) {
> +        /* There is only one datapath, so it should be moved out of the
> +         * group to a single 'od'. */
> +        size_t index = bitmap_scan(lflow->dpg_bitmap, true, 0,
> +                                    n_datapaths);
> +
> +        lflow->od = datapaths_array[index];
> +        lflow->dpg = NULL;
> +    } else {
> +        lflow->od = NULL;
> +    }
> +
> +    struct sbrec_logical_dp_group *sbrec_dp_group = NULL;
> +
> +    if (!sbflow) {
> +        lflow->sb_uuid = uuid_random();
> +        sbflow = sbrec_logical_flow_insert_persist_uuid(ovnsb_txn,
> +                                                        &lflow->sb_uuid);
> +        const char *pipeline = ovn_stage_get_pipeline_name(lflow->stage);
> +        uint8_t table = ovn_stage_get_table(lflow->stage);
> +        sbrec_logical_flow_set_pipeline(sbflow, pipeline);
> +        sbrec_logical_flow_set_table_id(sbflow, table);
> +        sbrec_logical_flow_set_priority(sbflow, lflow->priority);
> +        sbrec_logical_flow_set_match(sbflow, lflow->match);
> +        sbrec_logical_flow_set_actions(sbflow, lflow->actions);
> +        if (lflow->io_port) {
> +            struct smap tags = SMAP_INITIALIZER(&tags);
> +            smap_add(&tags, "in_out_port", lflow->io_port);
> +            sbrec_logical_flow_set_tags(sbflow, &tags);
> +            smap_destroy(&tags);
> +        }
> +        sbrec_logical_flow_set_controller_meter(sbflow, lflow->ctrl_meter);
> +
> +        /* Trim the source locator lflow->where, which looks something like
> +         * "ovn/northd/northd.c:1234", down to just the part following the
> +         * last slash, e.g. "northd.c:1234". */
> +        const char *slash = strrchr(lflow->where, '/');
> +#if _WIN32
> +        const char *backslash = strrchr(lflow->where, '\\');
> +        if (!slash || backslash > slash) {
> +            slash = backslash;
> +        }
> +#endif
> +        const char *where = slash ? slash + 1 : lflow->where;
> +
> +        struct smap ids = SMAP_INITIALIZER(&ids);
> +        smap_add(&ids, "stage-name", ovn_stage_to_str(lflow->stage));
> +        smap_add(&ids, "source", where);
> +        if (lflow->stage_hint) {
> +            smap_add(&ids, "stage-hint", lflow->stage_hint);
> +        }
> +        sbrec_logical_flow_set_external_ids(sbflow, &ids);
> +        smap_destroy(&ids);
> +
> +    } else {
> +        lflow->sb_uuid = sbflow->header_.uuid;
> +        sbrec_dp_group = sbflow->logical_dp_group;
> +
> +        if (lflow_input->ovn_internal_version_changed) {
> +            const char *stage_name = smap_get_def(&sbflow->external_ids,
> +                                                  "stage-name", "");
> +            const char *stage_hint = smap_get_def(&sbflow->external_ids,
> +                                                  "stage-hint", "");
> +            const char *source = smap_get_def(&sbflow->external_ids,
> +                                              "source", "");
> +
> +            if (strcmp(stage_name, ovn_stage_to_str(lflow->stage))) {
> +                sbrec_logical_flow_update_external_ids_setkey(
> +                    sbflow, "stage-name", ovn_stage_to_str(lflow->stage));
> +            }
> +            if (lflow->stage_hint) {
> +                if (strcmp(stage_hint, lflow->stage_hint)) {
> +                    sbrec_logical_flow_update_external_ids_setkey(
> +                        sbflow, "stage-hint", lflow->stage_hint);
> +                }
> +            }
> +            if (lflow->where) {
> +
> +                /* Trim the source locator lflow->where, which looks something
> +                 * like "ovn/northd/northd.c:1234", down to just the part
> +                 * following the last slash, e.g. "northd.c:1234". */
> +                const char *slash = strrchr(lflow->where, '/');
> +#if _WIN32
> +                const char *backslash = strrchr(lflow->where, '\\');
> +                if (!slash || backslash > slash) {
> +                    slash = backslash;
> +                }
> +#endif
> +                const char *where = slash ? slash + 1 : lflow->where;
> +
> +                if (strcmp(source, where)) {
> +                    sbrec_logical_flow_update_external_ids_setkey(
> +                        sbflow, "source", where);
> +                }
> +            }
> +        }
> +    }
> +
> +    if (lflow->od) {
> +        sbrec_logical_flow_set_logical_datapath(sbflow, lflow->od->sb);
> +        sbrec_logical_flow_set_logical_dp_group(sbflow, NULL);
> +    } else {
> +        sbrec_logical_flow_set_logical_datapath(sbflow, NULL);
> +        lflow->dpg = ovn_dp_group_get(dp_groups, lflow->n_ods,
> +                                      lflow->dpg_bitmap,
> +                                      n_datapaths);
> +        if (lflow->dpg) {
> +            /* Update the dpg's sb dp_group. */
> +            lflow->dpg->dp_group = sbrec_logical_dp_group_table_get_for_uuid(
> +                lflow_input->sbrec_logical_dp_group_table,
> +                &lflow->dpg->dpg_uuid);
> +            ovs_assert(lflow->dpg->dp_group);
> +        } else {
> +            lflow->dpg = ovn_dp_group_create(
> +                                ovnsb_txn, dp_groups, sbrec_dp_group,
> +                                lflow->n_ods, lflow->dpg_bitmap,
> +                                n_datapaths, is_switch,
> +                                lflow_input->ls_datapaths,
> +                                lflow_input->lr_datapaths);
> +        }
> +        sbrec_logical_flow_set_logical_dp_group(sbflow,
> +                                                lflow->dpg->dp_group);
> +    }
> +
> +    if (pre_sync_dpg != lflow->dpg) {
> +        if (lflow->dpg) {
> +            inc_ovn_dp_group_ref(lflow->dpg);
> +        }
> +        if (pre_sync_dpg) {
> +           dec_ovn_dp_group_ref(dp_groups, pre_sync_dpg);
> +        }
> +    }
> +}
> +
>  /* Updates the Logical_Flow and Multicast_Group tables in the OVN_SB database,
>   * constructing their contents based on the OVN_NB database. */
>  void build_lflows(struct ovsdb_idl_txn *ovnsb_txn,
>                    struct lflow_input *input_data,
> -                  struct hmap *lflows)
> +                  struct lflow_data *lflow_data)
>  {
>      struct hmap mcast_groups;
>      struct hmap igmp_groups;
> @@ -16556,7 +16916,7 @@ void build_lflows(struct ovsdb_idl_txn *ovnsb_txn,
>                                      input_data->lr_datapaths,
>                                      input_data->ls_ports,
>                                      input_data->lr_ports,
> -                                    input_data->port_groups, lflows,
> +                                    input_data->port_groups, lflow_data,
>                                      &igmp_groups,
>                                      input_data->meter_groups,
>                                      input_data->lb_datapaths_map,
> @@ -16571,72 +16931,17 @@ void build_lflows(struct ovsdb_idl_txn *ovnsb_txn,
>      /* Parallel build may result in a suboptimal hash. Resize the
>       * hash to a correct size before doing lookups */
>
> +    struct hmap *lflows = &lflow_data->lflows_match_map;
>      hmap_expand(lflows);
> +    hmap_expand(&lflow_data->lflows_hash_map);
>
>      if (hmap_count(lflows) > max_seen_lflow_size) {
>          max_seen_lflow_size = hmap_count(lflows);
>      }
>
> -    stopwatch_start(LFLOWS_DP_GROUPS_STOPWATCH_NAME, time_msec());
> -    /* Collecting all unique datapath groups. */
> -    struct hmap ls_dp_groups = HMAP_INITIALIZER(&ls_dp_groups);
> -    struct hmap lr_dp_groups = HMAP_INITIALIZER(&lr_dp_groups);
> -    struct hmap single_dp_lflows;
> -
> -    /* Single dp_flows will never grow bigger than lflows,
> -     * thus the two hmaps will remain the same size regardless
> -     * of how many elements we remove from lflows and add to
> -     * single_dp_lflows.
> -     * Note - lflows is always sized for at least 128 flows.
> -     */
> -    fast_hmap_size_for(&single_dp_lflows, max_seen_lflow_size);
> -
> -    struct ovn_lflow *lflow;
> -    HMAP_FOR_EACH_SAFE (lflow, hmap_node, lflows) {
> -        struct ovn_datapath **datapaths_array;
> -        size_t n_datapaths;
> -
> -        if (ovn_stage_to_datapath_type(lflow->stage) == DP_SWITCH) {
> -            n_datapaths = ods_size(input_data->ls_datapaths);
> -            datapaths_array = input_data->ls_datapaths->array;
> -        } else {
> -            n_datapaths = ods_size(input_data->lr_datapaths);
> -            datapaths_array = input_data->lr_datapaths->array;
> -        }
> -
> -        lflow->n_ods = bitmap_count1(lflow->dpg_bitmap, n_datapaths);
> -
> -        ovs_assert(lflow->n_ods);
> -
> -        if (lflow->n_ods == 1) {
> -            /* There is only one datapath, so it should be moved out of the
> -             * group to a single 'od'. */
> -            size_t index = bitmap_scan(lflow->dpg_bitmap, true, 0,
> -                                       n_datapaths);
> -
> -            bitmap_set0(lflow->dpg_bitmap, index);
> -            lflow->od = datapaths_array[index];
> -
> -            /* Logical flow should be re-hashed to allow lookups. */
> -            uint32_t hash = hmap_node_hash(&lflow->hmap_node);
> -            /* Remove from lflows. */
> -            hmap_remove(lflows, &lflow->hmap_node);
> -            hash = ovn_logical_flow_hash_datapath(&lflow->od->sb->header_.uuid,
> -                                                  hash);
> -            /* Add to single_dp_lflows. */
> -            hmap_insert_fast(&single_dp_lflows, &lflow->hmap_node, hash);
> -        }
> -    }
> -
> -    /* Merge multiple and single dp hashes. */
> -
> -    fast_hmap_merge(lflows, &single_dp_lflows);
> -
> -    hmap_destroy(&single_dp_lflows);
> -
> -    stopwatch_stop(LFLOWS_DP_GROUPS_STOPWATCH_NAME, time_msec());
>      stopwatch_start(LFLOWS_TO_SB_STOPWATCH_NAME, time_msec());
>
> +    struct ovn_lflow *lflow;
>      struct hmap lflows_temp = HMAP_INITIALIZER(&lflows_temp);
>      /* Push changes to the Logical_Flow table to database. */
>      const struct sbrec_logical_flow *sbflow;
> @@ -16680,68 +16985,15 @@ void build_lflows(struct ovsdb_idl_txn *ovnsb_txn,
>              = !strcmp(sbflow->pipeline, "ingress") ? P_IN : P_OUT;
>
>          lflow = ovn_lflow_find(
> -            lflows, dp_group ? NULL : logical_datapath_od,
> +            lflows,
>              ovn_stage_build(ovn_datapath_get_type(logical_datapath_od),
>                              pipeline, sbflow->table_id),
>              sbflow->priority, sbflow->match, sbflow->actions,
>              sbflow->controller_meter, sbflow->hash);
>          if (lflow) {
> -            struct hmap *dp_groups;
> -            size_t n_datapaths;
> -            bool is_switch;
> -
> -            lflow->sb_uuid = sbflow->header_.uuid;
> -            is_switch = ovn_stage_to_datapath_type(lflow->stage) == DP_SWITCH;
> -            if (is_switch) {
> -                n_datapaths = ods_size(input_data->ls_datapaths);
> -                dp_groups = &ls_dp_groups;
> -            } else {
> -                n_datapaths = ods_size(input_data->lr_datapaths);
> -                dp_groups = &lr_dp_groups;
> -            }
> -            if (input_data->ovn_internal_version_changed) {
> -                const char *stage_name = smap_get_def(&sbflow->external_ids,
> -                                                  "stage-name", "");
> -                const char *stage_hint = smap_get_def(&sbflow->external_ids,
> -                                                  "stage-hint", "");
> -                const char *source = smap_get_def(&sbflow->external_ids,
> -                                                  "source", "");
> +            sync_lflow_to_sb(ovnsb_txn, input_data, lflow_data,
> +                             lflow, sbflow);
>
> -                if (strcmp(stage_name, ovn_stage_to_str(lflow->stage))) {
> -                    sbrec_logical_flow_update_external_ids_setkey(sbflow,
> -                     "stage-name", ovn_stage_to_str(lflow->stage));
> -                }
> -                if (lflow->stage_hint) {
> -                    if (strcmp(stage_hint, lflow->stage_hint)) {
> -                        sbrec_logical_flow_update_external_ids_setkey(sbflow,
> -                        "stage-hint", lflow->stage_hint);
> -                    }
> -                }
> -                if (lflow->where) {
> -                    if (strcmp(source, lflow->where)) {
> -                        sbrec_logical_flow_update_external_ids_setkey(sbflow,
> -                        "source", lflow->where);
> -                    }
> -                }
> -            }
> -
> -            if (lflow->od) {
> -                sbrec_logical_flow_set_logical_datapath(sbflow, lflow->od->sb);
> -                sbrec_logical_flow_set_logical_dp_group(sbflow, NULL);
> -            } else {
> -                lflow->dpg = ovn_dp_group_get_or_create(
> -                                ovnsb_txn, dp_groups, dp_group,
> -                                lflow->n_ods, lflow->dpg_bitmap,
> -                                n_datapaths, is_switch,
> -                                input_data->ls_datapaths,
> -                                input_data->lr_datapaths);
> -
> -                sbrec_logical_flow_set_logical_datapath(sbflow, NULL);
> -                sbrec_logical_flow_set_logical_dp_group(sbflow,
> -                                                        lflow->dpg->dp_group);
> -            }
> -
> -            /* This lflow updated.  Not needed anymore. */
>              hmap_remove(lflows, &lflow->hmap_node);
>              hmap_insert(&lflows_temp, &lflow->hmap_node,
>                          hmap_node_hash(&lflow->hmap_node));
> @@ -16751,71 +17003,8 @@ void build_lflows(struct ovsdb_idl_txn *ovnsb_txn,
>      }
>
>      HMAP_FOR_EACH_SAFE (lflow, hmap_node, lflows) {
> -        const char *pipeline = ovn_stage_get_pipeline_name(lflow->stage);
> -        uint8_t table = ovn_stage_get_table(lflow->stage);
> -        struct hmap *dp_groups;
> -        size_t n_datapaths;
> -        bool is_switch;
> -
> -        is_switch = ovn_stage_to_datapath_type(lflow->stage) == DP_SWITCH;
> -        if (is_switch) {
> -            n_datapaths = ods_size(input_data->ls_datapaths);
> -            dp_groups = &ls_dp_groups;
> -        } else {
> -            n_datapaths = ods_size(input_data->lr_datapaths);
> -            dp_groups = &lr_dp_groups;
> -        }
> -
> -        lflow->sb_uuid = uuid_random();
> -        sbflow = sbrec_logical_flow_insert_persist_uuid(ovnsb_txn,
> -                                                        &lflow->sb_uuid);
> -        if (lflow->od) {
> -            sbrec_logical_flow_set_logical_datapath(sbflow, lflow->od->sb);
> -        } else {
> -            lflow->dpg = ovn_dp_group_get_or_create(
> -                                ovnsb_txn, dp_groups, NULL,
> -                                lflow->n_ods, lflow->dpg_bitmap,
> -                                n_datapaths, is_switch,
> -                                input_data->ls_datapaths,
> -                                input_data->lr_datapaths);
> -
> -            sbrec_logical_flow_set_logical_dp_group(sbflow,
> -                                                    lflow->dpg->dp_group);
> -        }
> -
> -        sbrec_logical_flow_set_pipeline(sbflow, pipeline);
> -        sbrec_logical_flow_set_table_id(sbflow, table);
> -        sbrec_logical_flow_set_priority(sbflow, lflow->priority);
> -        sbrec_logical_flow_set_match(sbflow, lflow->match);
> -        sbrec_logical_flow_set_actions(sbflow, lflow->actions);
> -        if (lflow->io_port) {
> -            struct smap tags = SMAP_INITIALIZER(&tags);
> -            smap_add(&tags, "in_out_port", lflow->io_port);
> -            sbrec_logical_flow_set_tags(sbflow, &tags);
> -            smap_destroy(&tags);
> -        }
> -        sbrec_logical_flow_set_controller_meter(sbflow, lflow->ctrl_meter);
> -
> -        /* Trim the source locator lflow->where, which looks something like
> -         * "ovn/northd/northd.c:1234", down to just the part following the
> -         * last slash, e.g. "northd.c:1234". */
> -        const char *slash = strrchr(lflow->where, '/');
> -#if _WIN32
> -        const char *backslash = strrchr(lflow->where, '\\');
> -        if (!slash || backslash > slash) {
> -            slash = backslash;
> -        }
> -#endif
> -        const char *where = slash ? slash + 1 : lflow->where;
> -
> -        struct smap ids = SMAP_INITIALIZER(&ids);
> -        smap_add(&ids, "stage-name", ovn_stage_to_str(lflow->stage));
> -        smap_add(&ids, "source", where);
> -        if (lflow->stage_hint) {
> -            smap_add(&ids, "stage-hint", lflow->stage_hint);
> -        }
> -        sbrec_logical_flow_set_external_ids(sbflow, &ids);
> -        smap_destroy(&ids);
> +        sync_lflow_to_sb(ovnsb_txn, input_data, lflow_data, lflow, NULL);
> +
>          hmap_remove(lflows, &lflow->hmap_node);
>          hmap_insert(&lflows_temp, &lflow->hmap_node,
>                      hmap_node_hash(&lflow->hmap_node));
> @@ -16824,17 +17013,6 @@ void build_lflows(struct ovsdb_idl_txn *ovnsb_txn,
>      hmap_destroy(&lflows_temp);
>
>      stopwatch_stop(LFLOWS_TO_SB_STOPWATCH_NAME, time_msec());
> -    struct ovn_dp_group *dpg;
> -    HMAP_FOR_EACH_POP (dpg, node, &ls_dp_groups) {
> -        bitmap_free(dpg->bitmap);
> -        free(dpg);
> -    }
> -    hmap_destroy(&ls_dp_groups);
> -    HMAP_FOR_EACH_POP (dpg, node, &lr_dp_groups) {
> -        bitmap_free(dpg->bitmap);
> -        free(dpg);
> -    }
> -    hmap_destroy(&lr_dp_groups);
>
>      /* Push changes to the Multicast_Group table to database. */
>      const struct sbrec_multicast_group *sbmc;
> @@ -16883,119 +17061,133 @@ void build_lflows(struct ovsdb_idl_txn *ovnsb_txn,
>      hmap_destroy(&mcast_groups);
>  }
>
> +static bool
> +is_lflow_and_od_type_match(struct ovn_datapath *od,
> +                           struct ovn_lflow *lflow)
> +{
> +    enum ovn_datapath_type type = od->nbs ? DP_SWITCH : DP_ROUTER;
> +    return ovn_stage_to_datapath_type(lflow->stage) == type;
> +}
> +
>  static void
> -sync_lsp_lflows_to_sb(struct ovsdb_idl_txn *ovnsb_txn,
> -                      struct lflow_input *lflow_input,
> -                      struct hmap *lflows,
> -                      struct ovn_lflow *lflow)
> +unlink_objres_lflows(struct resource_to_objects_node  *res_node,
> +                     struct ovn_datapath *od,
> +                     struct lflow_data *lflow_data,
> +                     struct objdep_mgr *lflowdep_mgr)
>  {
> -    size_t n_datapaths;
> -    struct ovn_datapath **datapaths_array;
> -    if (ovn_stage_to_datapath_type(lflow->stage) == DP_SWITCH) {
> -        n_datapaths = ods_size(lflow_input->ls_datapaths);
> -        datapaths_array = lflow_input->ls_datapaths->array;
> -    } else {
> -        n_datapaths = ods_size(lflow_input->lr_datapaths);
> -        datapaths_array = lflow_input->lr_datapaths->array;
> -    }
> -    uint32_t n_ods = bitmap_count1(lflow->dpg_bitmap, n_datapaths);
> -    ovs_assert(n_ods == 1);
> -    /* There is only one datapath, so it should be moved out of the
> -     * group to a single 'od'. */
> -    size_t index = bitmap_scan(lflow->dpg_bitmap, true, 0,
> -                               n_datapaths);
> -
> -    bitmap_set0(lflow->dpg_bitmap, index);
> -    lflow->od = datapaths_array[index];
> -
> -    /* Logical flow should be re-hashed to allow lookups. */
> -    uint32_t hash = hmap_node_hash(&lflow->hmap_node);
> -    /* Remove from lflows. */
> -    hmap_remove(lflows, &lflow->hmap_node);
> -    hash = ovn_logical_flow_hash_datapath(&lflow->od->sb->header_.uuid,
> -                                          hash);
> -    /* Add back. */
> -    hmap_insert(lflows, &lflow->hmap_node, hash);
> -
> -    /* Sync to SB. */
> -    const struct sbrec_logical_flow *sbflow;
> -    /* Note: uuid_random acquires a global mutex. If we parallelize the sync to
> -     * SB this may become a bottleneck. */
> -    lflow->sb_uuid = uuid_random();
> -    sbflow = sbrec_logical_flow_insert_persist_uuid(ovnsb_txn,
> -                                                    &lflow->sb_uuid);
> -    const char *pipeline = ovn_stage_get_pipeline_name(lflow->stage);
> -    uint8_t table = ovn_stage_get_table(lflow->stage);
> -    sbrec_logical_flow_set_logical_datapath(sbflow, lflow->od->sb);
> -    sbrec_logical_flow_set_logical_dp_group(sbflow, NULL);
> -    sbrec_logical_flow_set_pipeline(sbflow, pipeline);
> -    sbrec_logical_flow_set_table_id(sbflow, table);
> -    sbrec_logical_flow_set_priority(sbflow, lflow->priority);
> -    sbrec_logical_flow_set_match(sbflow, lflow->match);
> -    sbrec_logical_flow_set_actions(sbflow, lflow->actions);
> -    if (lflow->io_port) {
> -        struct smap tags = SMAP_INITIALIZER(&tags);
> -        smap_add(&tags, "in_out_port", lflow->io_port);
> -        sbrec_logical_flow_set_tags(sbflow, &tags);
> -        smap_destroy(&tags);
> -    }
> -    sbrec_logical_flow_set_controller_meter(sbflow, lflow->ctrl_meter);
> -    /* Trim the source locator lflow->where, which looks something like
> -     * "ovn/northd/northd.c:1234", down to just the part following the
> -     * last slash, e.g. "northd.c:1234". */
> -    const char *slash = strrchr(lflow->where, '/');
> -#if _WIN32
> -    const char *backslash = strrchr(lflow->where, '\\');
> -    if (!slash || backslash > slash) {
> -        slash = backslash;
> +    if (!res_node) {
> +        return;
>      }
> -#endif
> -    const char *where = slash ? slash + 1 : lflow->where;
>
> -    struct smap ids = SMAP_INITIALIZER(&ids);
> -    smap_add(&ids, "stage-name", ovn_stage_to_str(lflow->stage));
> -    smap_add(&ids, "source", where);
> -    if (lflow->stage_hint) {
> -        smap_add(&ids, "stage-hint", lflow->stage_hint);
> +    struct object_to_resources_list_node *resource_list_node;
> +    RESOURCE_FOR_EACH_OBJ (resource_list_node, res_node) {
> +        const struct uuid *obj_uuid = &resource_list_node->obj_uuid;
> +        struct ovn_lflow *lflow = ovn_lflow_uuid_find(
> +            &lflow_data->lflows_hash_map, obj_uuid);
> +        if (!lflow) {
> +            continue;
> +        }
> +
> +
> +        /* Check if the lflow datapath is same the od datapath. */
> +        if (is_lflow_and_od_type_match(od, lflow)) {
> +            bitmap_set0(lflow->dpg_bitmap, od->index);
> +        }  else {
> +            /* The datapath type doesn't match.  Which means this lflow was
> +             * added due to a resource in the other type.
> +             * Eg. For every logical switch port whose lswitch is connected
> +             * to a router, an lflow is added in the lr_in_arp_resolve stage.
> +             * Get the datapath index of this router and clear it.
> +             *
> +             * OBJDEP_TYPE_LFLOW_OD type is used to store this lflow object to
> +             * logical router resource linking (logical router index is stored
> +             * in the uuid.parts[0]).
> +             */
> +            char uuid_s[UUID_LEN + 1];
> +            sprintf(uuid_s, UUID_FMT, UUID_ARGS(&lflow->lflow_uuid));
> +
> +            struct resource_to_objects_node  *lflow_od_res =
> +                objdep_mgr_find_objs(lflowdep_mgr, OBJDEP_TYPE_LFLOW_OD,
> +                                     uuid_s);
> +            if (lflow_od_res) {
> +                struct object_to_resources_list_node *r_node;
> +                RESOURCE_FOR_EACH_OBJ (r_node, lflow_od_res) {
> +                    size_t index = r_node->obj_uuid.parts[0];
> +                    bitmap_set0(lflow->dpg_bitmap, index);
> +                }
> +            }
> +        }
>      }
> -    sbrec_logical_flow_set_external_ids(sbflow, &ids);
> -    smap_destroy(&ids);
>  }
>
>  static bool
> -delete_lflow_for_lsp(struct ovn_port *op, bool is_update,
> -                     const struct sbrec_logical_flow_table *sb_lflow_table,
> -                     struct hmap *lflows)
> +sync_lflows_from_objres(struct ovsdb_idl_txn *ovnsb_txn,
> +                        struct resource_to_objects_node  *res_node,
> +                        struct lflow_input *lflow_input,
> +                        struct lflow_data *lflow_data,
> +                        struct objdep_mgr *lflowdep_mgr)
>  {
> -    struct lflow_ref_node *lfrn;
> -    const char *operation = is_update ? "updated" : "deleted";
> -    LIST_FOR_EACH_SAFE (lfrn, lflow_list_node, &op->lflows) {
> -        VLOG_DBG("Deleting SB lflow "UUID_FMT" for %s port %s",
> -                 UUID_ARGS(&lfrn->lflow->sb_uuid), operation, op->key);
> +    if (!res_node) {
> +        return true;
> +    }
> +
> +    struct uuidset lflow_uuidset = UUIDSET_INITIALIZER(&lflow_uuidset);
> +    struct object_to_resources_list_node *resource_list_node;
> +    RESOURCE_FOR_EACH_OBJ (resource_list_node, res_node) {
> +        const struct uuid *obj_uuid = &resource_list_node->obj_uuid;
> +
> +        struct ovn_lflow *lflow = ovn_lflow_uuid_find(
> +            &lflow_data->lflows_hash_map, obj_uuid);
> +        if (!lflow) {
> +            continue;
> +        }
>
>          const struct sbrec_logical_flow *sblflow =
> -            sbrec_logical_flow_table_get_for_uuid(sb_lflow_table,
> -                                              &lfrn->lflow->sb_uuid);
> -        if (sblflow) {
> -            sbrec_logical_flow_delete(sblflow);
> +            sbrec_logical_flow_table_get_for_uuid(
> +                lflow_input->sbrec_logical_flow_table, &lflow->sb_uuid);
> +
> +        size_t n_datapaths;
> +        if (ovn_stage_to_datapath_type(lflow->stage) == DP_SWITCH) {
> +            n_datapaths = ods_size(lflow_input->ls_datapaths);
>          } else {
> -            static struct vlog_rate_limit rl =
> -                VLOG_RATE_LIMIT_INIT(1, 1);
> -            VLOG_WARN_RL(&rl, "SB lflow "UUID_FMT" not found when handling "
> -                         "%s port %s. Recompute.",
> -                         UUID_ARGS(&lfrn->lflow->sb_uuid), operation, op->key);
> -            return false;
> +            n_datapaths = ods_size(lflow_input->lr_datapaths);
>          }
>
> -        ovn_lflow_destroy(lflows, lfrn->lflow);
> +        size_t n_ods = bitmap_count1(lflow->dpg_bitmap, n_datapaths);
> +
> +        if (n_ods) {
> +            sync_lflow_to_sb(ovnsb_txn, lflow_input, lflow_data,
> +                             lflow, sblflow);
> +        } else {
> +            if (sblflow) {
> +                sbrec_logical_flow_delete(sblflow);

Please note that there is a bug here.  Call to "ovn_flow_destroy() is
missing after deleting the SB lflow.

I'll fix it in v6.

Thanks
Numan

> +            } else {
> +                static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(1, 1);
> +                VLOG_WARN_RL(&rl, "SB lflow "UUID_FMT" not found when "
> +                            "deleting lflows for resource %s (type %d). "
> +                            "Recompute.",
> +                            UUID_ARGS(&lflow->sb_uuid),
> +                            resource_list_node->resource_node->res_name,
> +                            resource_list_node->resource_node->type);
> +                return false;
> +            }
> +            uuidset_insert(&lflow_uuidset, obj_uuid);
> +        }
> +    }
> +
> +    struct uuidset_node *unode;
> +    UUIDSET_FOR_EACH (unode, &lflow_uuidset) {
> +        objdep_mgr_remove_obj(lflowdep_mgr, &unode->uuid);
>      }
> +    uuidset_destroy(&lflow_uuidset);
> +
>      return true;
>  }
>
>  bool lflow_handle_northd_ls_changes(struct ovsdb_idl_txn *ovnsb_txn,
>                                      struct tracked_ls_changes *ls_changes,
>                                      struct lflow_input *lflow_input,
> -                                    struct hmap *lflows)
> +                                    struct lflow_data *lflow_data)
>  {
>      struct ls_change *ls_change;
>
> @@ -17012,23 +17204,29 @@ bool lflow_handle_northd_ls_changes(struct ovsdb_idl_txn *ovnsb_txn,
>
>          struct ovn_port *op;
>          LIST_FOR_EACH (op, list, &ls_change->deleted_ports) {
> -            if (!delete_lflow_for_lsp(op, false,
> -                                      lflow_input->sbrec_logical_flow_table,
> -                                      lflows)) {
> -                return false;
> -            }
> +            struct resource_to_objects_node  *res_node =
> +                objdep_mgr_find_objs(&op->lflow_dep_mgr, OBJDEP_TYPE_LPORT,
> +                                     op->nbsp->name);
> +
> +            /* unlink old lflows. */
> +            unlink_objres_lflows(res_node, op->od, lflow_data,
> +                                 &op->lflow_dep_mgr);
> +            sync_lflows_from_objres(ovnsb_txn, res_node, lflow_input,
> +                                    lflow_data, &op->lflow_dep_mgr);
> +            objdep_mgr_clear(&op->lflow_dep_mgr);
>
>              /* No need to update SB multicast groups, thanks to weak
>               * references. */
>          }
>
>          LIST_FOR_EACH (op, list, &ls_change->updated_ports) {
> -            /* Delete old lflows. */
> -            if (!delete_lflow_for_lsp(op, true,
> -                                      lflow_input->sbrec_logical_flow_table,
> -                                      lflows)) {
> -                return false;
> -            }
> +            struct resource_to_objects_node  *res_node =
> +                objdep_mgr_find_objs(&op->lflow_dep_mgr, OBJDEP_TYPE_LPORT,
> +                                     op->nbsp->name);
> +
> +            /* unlink old lflows. */
> +            unlink_objres_lflows(res_node, op->od,
> +                                 lflow_data, &op->lflow_dep_mgr);
>
>              /* Generate new lflows. */
>              struct ds match = DS_EMPTY_INITIALIZER;
> @@ -17037,7 +17235,7 @@ bool lflow_handle_northd_ls_changes(struct ovsdb_idl_txn *ovnsb_txn,
>                                                       lflow_input->lr_ports,
>                                                       lflow_input->meter_groups,
>                                                       &match, &actions,
> -                                                     lflows);
> +                                                     lflow_data);
>              ds_destroy(&match);
>              ds_destroy(&actions);
>
> @@ -17045,11 +17243,11 @@ bool lflow_handle_northd_ls_changes(struct ovsdb_idl_txn *ovnsb_txn,
>               * groups. */
>
>              /* Sync the new flows to SB. */
> -            struct lflow_ref_node *lfrn;
> -            LIST_FOR_EACH (lfrn, lflow_list_node, &op->lflows) {
> -                sync_lsp_lflows_to_sb(ovnsb_txn, lflow_input, lflows,
> -                                      lfrn->lflow);
> -            }
> +            res_node = objdep_mgr_find_objs(&op->lflow_dep_mgr,
> +                                            OBJDEP_TYPE_LPORT,
> +                                            op->nbsp->name);
> +            sync_lflows_from_objres(ovnsb_txn, res_node, lflow_input,
> +                                    lflow_data, &op->lflow_dep_mgr);
>          }
>
>          LIST_FOR_EACH (op, list, &ls_change->added_ports) {
> @@ -17059,7 +17257,7 @@ bool lflow_handle_northd_ls_changes(struct ovsdb_idl_txn *ovnsb_txn,
>                                                       lflow_input->lr_ports,
>                                                       lflow_input->meter_groups,
>                                                       &match, &actions,
> -                                                     lflows);
> +                                                     lflow_data);
>              ds_destroy(&match);
>              ds_destroy(&actions);
>
> @@ -17088,11 +17286,12 @@ bool lflow_handle_northd_ls_changes(struct ovsdb_idl_txn *ovnsb_txn,
>              }
>
>              /* Sync the newly added flows to SB. */
> -            struct lflow_ref_node *lfrn;
> -            LIST_FOR_EACH (lfrn, lflow_list_node, &op->lflows) {
> -                sync_lsp_lflows_to_sb(ovnsb_txn, lflow_input, lflows,
> -                                      lfrn->lflow);
> -            }
> +            struct resource_to_objects_node  *res_node;
> +            res_node = objdep_mgr_find_objs(&op->lflow_dep_mgr,
> +                                            OBJDEP_TYPE_LPORT,
> +                                            op->nbsp->name);
> +            sync_lflows_from_objres(ovnsb_txn, res_node, lflow_input,
> +                                    lflow_data, &op->lflow_dep_mgr);
>          }
>      }
>      return true;
> diff --git a/northd/northd.h b/northd/northd.h
> index 1d344d57d6..4af6c4e9b0 100644
> --- a/northd/northd.h
> +++ b/northd/northd.h
> @@ -19,6 +19,7 @@
>  #include "lib/ovn-sb-idl.h"
>  #include "lib/ovn-util.h"
>  #include "lib/ovs-atomic.h"
> +#include "lib/objdep.h"
>  #include "lib/sset.h"
>  #include "northd/ipam.h"
>  #include "openvswitch/hmap.h"
> @@ -125,7 +126,10 @@ struct northd_data {
>  };
>
>  struct lflow_data {
> -    struct hmap lflows;
> +    struct hmap lflows_match_map;
> +    struct hmap lflows_hash_map;
> +    struct hmap ls_dp_groups;
> +    struct hmap lr_dp_groups;
>  };
>
>  void lflow_data_init(struct lflow_data *);
> @@ -140,6 +144,7 @@ struct lflow_input {
>      const struct sbrec_logical_flow_table *sbrec_logical_flow_table;
>      const struct sbrec_multicast_group_table *sbrec_multicast_group_table;
>      const struct sbrec_igmp_group_table *sbrec_igmp_group_table;
> +    const struct sbrec_logical_dp_group_table *sbrec_logical_dp_group_table;
>
>      /* Indexes */
>      struct ovsdb_idl_index *sbrec_mcast_group_by_name_dp;
> @@ -344,10 +349,10 @@ void northd_indices_create(struct northd_data *data,
>                             struct ovsdb_idl *ovnsb_idl);
>  void build_lflows(struct ovsdb_idl_txn *ovnsb_txn,
>                    struct lflow_input *input_data,
> -                  struct hmap *lflows);
> +                  struct lflow_data *lflow_data);
>  bool lflow_handle_northd_ls_changes(struct ovsdb_idl_txn *ovnsb_txn,
>                                      struct tracked_ls_changes *,
> -                                    struct lflow_input *, struct hmap *lflows);
> +                                    struct lflow_input *, struct lflow_data *);
>  bool northd_handle_sb_port_binding_changes(
>      const struct sbrec_port_binding_table *, struct hmap *ls_ports);
>
> diff --git a/northd/ovn-northd.c b/northd/ovn-northd.c
> index 4fa1b039ea..edca0552c0 100644
> --- a/northd/ovn-northd.c
> +++ b/northd/ovn-northd.c
> @@ -836,6 +836,10 @@ main(int argc, char *argv[])
>          ovsdb_idl_omit_alert(ovnsb_idl_loop.idl,
>                               &sbrec_multicast_group_columns[i]);
>      }
> +    for (size_t i = 0; i < SBREC_LOGICAL_DP_GROUP_N_COLUMNS; i++) {
> +        ovsdb_idl_omit_alert(ovnsb_idl_loop.idl,
> +                             &sbrec_logical_dp_group_columns[i]);
> +    }
>
>      unixctl_command_register("sb-connection-status", "", 0, 0,
>                               ovn_conn_show, ovnsb_idl_loop.idl);
> --
> 2.40.1
>
> _______________________________________________
> dev mailing list
> dev@openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-dev
>

On Wed, Aug 9, 2023 at 9:38 AM <numans@ovn.org> wrote:
>
> From: Numan Siddique <numans@ovn.org>
>
Thanks Numan for the enhancement.

> Instead of maintaing the lport to lflow references using
> 'struct lflow_ref_list', this patch makes use of the existing
> objdep APIs.  Since objdep_mgr is not thread safe, an instance
> of objdep_mgr is maintained for each 'ovn_port'.
>
> Once we add the thread safe support to objdep APIs we can move
> the objdep_mgr to 'lflow' engine date.
>
> Using objdep APIs does come with a cost in memory.  We now
> maintain an additional hmap of ovn_lflow's to look up using
> uuid.  But this will be useful to handle datapath and load
> balancer changes in the lflow engine node.
>

I understand that the 'struct lflow_ref_list' may not satisfy the needs of
more complex I-P, but I wonder if objdep_mgr (without modification) is the
solution. It seems to me more complex at least for this patch without
adding obvious benefits. Since I am still reviewing, maybe I missed some
points which would show the value from the rest of the patches, and please
forgive me for my slow review.

While I am still reviewing the patches, I did a performance test with
https://github.com/hzhou8/ovn-test-script for the 200 chassis x 50 lsp
scale of simulated ovn-k8s topology.

- Before this patch, a recompute takes 1293ms
- After this patch, a recompute takes 1536ms, which is 20% more time.
- I also did the same test with all the rest of the patches of this series,
which takes 1690ms.

So the major increase of the recompute time is from this patch. I didn't
debug what's the exact cause of this increase, but I think it might be
related to the objdep usage.

> This patch does few more changes which are significant:
>   -  Logical flows are now hashed without the logical
>      datapaths.  If a logical flow is referenced by just one
>      datapath, we don't rehash it.
>
>   -  The synthetic 'hash' column of sbrec_logical_flow now
>      doesn't use the logical datapath too.  This means that
>      when ovn-northd is updated/upgraded and has this commit,
>      all the logical flows with 'logical_datapath' column
>      set will get deleted and re-added causing some disruptions.
>
>   -  With the commit [1] which added I-P support for logical
>      port changes, multiple logical flows with same match 'M'
>      and actions 'A' are generated and stored without the
>      dp groups, which was not the case prior to
>      that patch.
>      One example to generate these lflows is:
>              ovn-nbctl lsp-set-addresses sw0p1 "MAC1 IP1"
>              ovn-nbctl lsp-set-addresses sw1p1 "MAC1 IP1"
>              ovn-nbctl lsp-set-addresses sw2p1 "MAC1 IP1"
>
>      Now with this patch we go back to the earlier way.  i.e
>      one logical flow with logical_dp_groups set.
>
Thanks for taking the effort to achieve this. The approach is indeed very
smart. I was aware of such scenarios, but it just didn''t seem to impact
scalability because the number of such lflows should be small, so it wasn't
considered a priority.

>   -  With this patch any updates to a logical port which
>      doesn't result in new logical flows will not result in
>      deletion and addition of same logical flows.
>      Eg.
>      ovn-nbctl set logical_switch_port sw0p1 external_ids:foo=bar
>      will be a no-op to the SB logical flow table.
>
Similar to the above case, this doesn't seem to be critical for
scalability, but it is good to see the enhancement.
I will post more feedback after finishing the full review.

Thanks,
Han

> [1] - 8bbd67891f68("northd: Incremental processing of VIF additions in
'lflow' node.")
>
> Suggested-by: Dumitru Ceara <dceara@redhat.com>
> Signed-off-by: Numan Siddique <numans@ovn.org>

On Fri, Aug 18, 2023 at 11:38 AM Han Zhou <hzhou@ovn.org> wrote:
>
> On Wed, Aug 9, 2023 at 9:38 AM <numans@ovn.org> wrote:
> >
> > From: Numan Siddique <numans@ovn.org>
> >
> Thanks Numan for the enhancement.
>
> > Instead of maintaing the lport to lflow references using
> > 'struct lflow_ref_list', this patch makes use of the existing
> > objdep APIs.  Since objdep_mgr is not thread safe, an instance
> > of objdep_mgr is maintained for each 'ovn_port'.
> >
> > Once we add the thread safe support to objdep APIs we can move
> > the objdep_mgr to 'lflow' engine date.
> >
> > Using objdep APIs does come with a cost in memory.  We now
> > maintain an additional hmap of ovn_lflow's to look up using
> > uuid.  But this will be useful to handle datapath and load
> > balancer changes in the lflow engine node.
> >
>
> I understand that the 'struct lflow_ref_list' may not satisfy the needs of
> more complex I-P, but I wonder if objdep_mgr (without modification) is the
> solution. It seems to me more complex at least for this patch without
> adding obvious benefits. Since I am still reviewing, maybe I missed some
> points which would show the value from the rest of the patches, and please
> forgive me for my slow review.

Sure.   No worries.

Can you please continue reviewing from the latest v6 ?

http://patchwork.ozlabs.org/project/ovn/list/?series=369384

I think we can discuss more once you go through all the patches in the
series (or until patch 14).
I think that would give a better idea of how it is used and we can
discuss if it's worth using obj dep mgr.

Also I think the patches P1 - P8 make one set and can be considered
first.  I submitted all these patches
as one series mainly because
   -  all the patches can be applied easily
   -  the soft freeze was near.

The patches can be found here too -
https://github.com/numansiddique/ovn/tree/northd_ip_lb_ip_v6

Thanks
Numan

>
> While I am still reviewing the patches, I did a performance test with
> https://github.com/hzhou8/ovn-test-script for the 200 chassis x 50 lsp
> scale of simulated ovn-k8s topology.
>
> - Before this patch, a recompute takes 1293ms
> - After this patch, a recompute takes 1536ms, which is 20% more time.
> - I also did the same test with all the rest of the patches of this series,
> which takes 1690ms.
>
> So the major increase of the recompute time is from this patch. I didn't
> debug what's the exact cause of this increase, but I think it might be
> related to the objdep usage.
>
> > This patch does few more changes which are significant:
> >   -  Logical flows are now hashed without the logical
> >      datapaths.  If a logical flow is referenced by just one
> >      datapath, we don't rehash it.
> >
> >   -  The synthetic 'hash' column of sbrec_logical_flow now
> >      doesn't use the logical datapath too.  This means that
> >      when ovn-northd is updated/upgraded and has this commit,
> >      all the logical flows with 'logical_datapath' column
> >      set will get deleted and re-added causing some disruptions.
> >
> >   -  With the commit [1] which added I-P support for logical
> >      port changes, multiple logical flows with same match 'M'
> >      and actions 'A' are generated and stored without the
> >      dp groups, which was not the case prior to
> >      that patch.
> >      One example to generate these lflows is:
> >              ovn-nbctl lsp-set-addresses sw0p1 "MAC1 IP1"
> >              ovn-nbctl lsp-set-addresses sw1p1 "MAC1 IP1"
> >              ovn-nbctl lsp-set-addresses sw2p1 "MAC1 IP1"
> >
> >      Now with this patch we go back to the earlier way.  i.e
> >      one logical flow with logical_dp_groups set.
> >
> Thanks for taking the effort to achieve this. The approach is indeed very
> smart. I was aware of such scenarios, but it just didn''t seem to impact
> scalability because the number of such lflows should be small, so it wasn't
> considered a priority.
>
> >   -  With this patch any updates to a logical port which
> >      doesn't result in new logical flows will not result in
> >      deletion and addition of same logical flows.
> >      Eg.
> >      ovn-nbctl set logical_switch_port sw0p1 external_ids:foo=bar
> >      will be a no-op to the SB logical flow table.
> >
> Similar to the above case, this doesn't seem to be critical for
> scalability, but it is good to see the enhancement.
> I will post more feedback after finishing the full review.
>
> Thanks,
> Han
>
> > [1] - 8bbd67891f68("northd: Incremental processing of VIF additions in
> 'lflow' node.")
> >
> > Suggested-by: Dumitru Ceara <dceara@redhat.com>
> > Signed-off-by: Numan Siddique <numans@ovn.org>
> _______________________________________________
> dev mailing list
> dev@openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-dev

On Fri, Aug 18, 2023 at 2:33 PM Numan Siddique <numans@ovn.org> wrote:
>
> On Fri, Aug 18, 2023 at 11:38 AM Han Zhou <hzhou@ovn.org> wrote:
> >
> > On Wed, Aug 9, 2023 at 9:38 AM <numans@ovn.org> wrote:
> > >
> > > From: Numan Siddique <numans@ovn.org>
> > >
> > Thanks Numan for the enhancement.
> >
> > > Instead of maintaing the lport to lflow references using
> > > 'struct lflow_ref_list', this patch makes use of the existing
> > > objdep APIs.  Since objdep_mgr is not thread safe, an instance
> > > of objdep_mgr is maintained for each 'ovn_port'.
> > >
> > > Once we add the thread safe support to objdep APIs we can move
> > > the objdep_mgr to 'lflow' engine date.
> > >
> > > Using objdep APIs does come with a cost in memory.  We now
> > > maintain an additional hmap of ovn_lflow's to look up using
> > > uuid.  But this will be useful to handle datapath and load
> > > balancer changes in the lflow engine node.
> > >
> >
> > I understand that the 'struct lflow_ref_list' may not satisfy the needs of
> > more complex I-P, but I wonder if objdep_mgr (without modification) is the
> > solution. It seems to me more complex at least for this patch without
> > adding obvious benefits. Since I am still reviewing, maybe I missed some
> > points which would show the value from the rest of the patches, and please
> > forgive me for my slow review.
>
> Sure.   No worries.
>
> Can you please continue reviewing from the latest v6 ?
>
> http://patchwork.ozlabs.org/project/ovn/list/?series=369384
>
> I think we can discuss more once you go through all the patches in the
> series (or until patch 14).
> I think that would give a better idea of how it is used and we can
> discuss if it's worth using obj dep mgr.
>
> Also I think the patches P1 - P8 make one set and can be considered
> first.  I submitted all these patches
> as one series mainly because
>    -  all the patches can be applied easily
>    -  the soft freeze was near.
>
> The patches can be found here too -
> https://github.com/numansiddique/ovn/tree/northd_ip_lb_ip_v6
>
> Thanks
> Numan
>
> >
> > While I am still reviewing the patches, I did a performance test with
> > https://github.com/hzhou8/ovn-test-script for the 200 chassis x 50 lsp
> > scale of simulated ovn-k8s topology.
> >
> > - Before this patch, a recompute takes 1293ms
> > - After this patch, a recompute takes 1536ms, which is 20% more time.
> > - I also did the same test with all the rest of the patches of this series,
> > which takes 1690ms.
> >

I did some testing using the OVN dbs of  the ovn-heater density heavy run.
A triggered forced recompute with the present main of ovn-northd takes
around 4389ms
where as with the patch 9 of this series,  it is taking around 4838ms.

I hacked the 'engine_compute_log_timeout_msec' [1] value from 500ms to
50ms so that
inc-eng can log the engine nodes taking more than 50ms in the recompute.

With main
-------
2023-08-18T11:28:20.714Z|00117|inc_proc_eng|INFO|User triggered force recompute.
2023-08-18T11:28:23.189Z|00118|inc_proc_eng|INFO|node: northd,
recompute (forced) took 2475ms
2023-08-18T11:28:25.102Z|00119|inc_proc_eng|INFO|node: lflow,
recompute (forced) took 1887ms
2023-08-18T11:28:25.102Z|00120|timeval|WARN|Unreasonably long 4389ms
poll interval (4373ms user, 3ms system)


And with patch 9
---------------------
2023-08-18T11:28:10.558Z|00220|inc_proc_eng|INFO|User triggered force recompute.
2023-08-18T11:28:10.883Z|00221|inc_proc_eng|INFO|node: lb_data,
recompute (forced) took 325ms
2023-08-18T11:28:13.163Z|00222|inc_proc_eng|INFO|node: northd,
recompute (forced) took 2280ms
2023-08-18T11:28:13.431Z|00223|inc_proc_eng|INFO|node: sync_to_sb_lb,
recompute (forced) took 241ms
2023-08-18T11:28:15.396Z|00224|inc_proc_eng|INFO|node: lflow,
recompute (forced) took 1956ms
2023-08-18T11:28:15.397Z|00225|timeval|WARN|Unreasonably long 4838ms
poll interval (4819ms user, 3ms system)


As you can see the lflow recompute time is not very significant
(1887 ms vs 1956 ms)

It is the lb_data engine node which is taking 325ms more.  But if you
see the lb_data engine code, the lb_data handlers never return false.
So the lb_data full engine recompute cost can be ignored as it will be
triggered when engine aborts or when the user triggers a recompute.

I don't think the lflow engine recompute cost due to objdep is
significant if lflow recomputes are less.

Thanks
Numan

> > So the major increase of the recompute time is from this patch. I didn't
> > debug what's the exact cause of this increase, but I think it might be
> > related to the objdep usage.
> >
> > > This patch does few more changes which are significant:
> > >   -  Logical flows are now hashed without the logical
> > >      datapaths.  If a logical flow is referenced by just one
> > >      datapath, we don't rehash it.
> > >
> > >   -  The synthetic 'hash' column of sbrec_logical_flow now
> > >      doesn't use the logical datapath too.  This means that
> > >      when ovn-northd is updated/upgraded and has this commit,
> > >      all the logical flows with 'logical_datapath' column
> > >      set will get deleted and re-added causing some disruptions.
> > >
> > >   -  With the commit [1] which added I-P support for logical
> > >      port changes, multiple logical flows with same match 'M'
> > >      and actions 'A' are generated and stored without the
> > >      dp groups, which was not the case prior to
> > >      that patch.
> > >      One example to generate these lflows is:
> > >              ovn-nbctl lsp-set-addresses sw0p1 "MAC1 IP1"
> > >              ovn-nbctl lsp-set-addresses sw1p1 "MAC1 IP1"
> > >              ovn-nbctl lsp-set-addresses sw2p1 "MAC1 IP1"
> > >
> > >      Now with this patch we go back to the earlier way.  i.e
> > >      one logical flow with logical_dp_groups set.
> > >
> > Thanks for taking the effort to achieve this. The approach is indeed very
> > smart. I was aware of such scenarios, but it just didn''t seem to impact
> > scalability because the number of such lflows should be small, so it wasn't
> > considered a priority.
> >
> > >   -  With this patch any updates to a logical port which
> > >      doesn't result in new logical flows will not result in
> > >      deletion and addition of same logical flows.
> > >      Eg.
> > >      ovn-nbctl set logical_switch_port sw0p1 external_ids:foo=bar
> > >      will be a no-op to the SB logical flow table.
> > >
> > Similar to the above case, this doesn't seem to be critical for
> > scalability, but it is good to see the enhancement.
> > I will post more feedback after finishing the full review.
> >
> > Thanks,
> > Han
> >
> > > [1] - 8bbd67891f68("northd: Incremental processing of VIF additions in
> > 'lflow' node.")
> > >
> > > Suggested-by: Dumitru Ceara <dceara@redhat.com>
> > > Signed-off-by: Numan Siddique <numans@ovn.org>
> > _______________________________________________
> > dev mailing list
> > dev@openvswitch.org
> > https://mail.openvswitch.org/mailman/listinfo/ovs-dev

On Fri, Aug 18, 2023 at 4:35 AM Numan Siddique <numans@ovn.org> wrote:
>
> On Fri, Aug 18, 2023 at 2:33 PM Numan Siddique <numans@ovn.org> wrote:
> >
> > On Fri, Aug 18, 2023 at 11:38 AM Han Zhou <hzhou@ovn.org> wrote:
> > >
> > > On Wed, Aug 9, 2023 at 9:38 AM <numans@ovn.org> wrote:
> > > >
> > > > From: Numan Siddique <numans@ovn.org>
> > > >
> > > Thanks Numan for the enhancement.
> > >
> > > > Instead of maintaing the lport to lflow references using
> > > > 'struct lflow_ref_list', this patch makes use of the existing
> > > > objdep APIs.  Since objdep_mgr is not thread safe, an instance
> > > > of objdep_mgr is maintained for each 'ovn_port'.
> > > >
> > > > Once we add the thread safe support to objdep APIs we can move
> > > > the objdep_mgr to 'lflow' engine date.
> > > >
> > > > Using objdep APIs does come with a cost in memory.  We now
> > > > maintain an additional hmap of ovn_lflow's to look up using
> > > > uuid.  But this will be useful to handle datapath and load
> > > > balancer changes in the lflow engine node.
> > > >
> > >
> > > I understand that the 'struct lflow_ref_list' may not satisfy the
needs of
> > > more complex I-P, but I wonder if objdep_mgr (without modification)
is the
> > > solution. It seems to me more complex at least for this patch without
> > > adding obvious benefits. Since I am still reviewing, maybe I missed
some
> > > points which would show the value from the rest of the patches, and
please
> > > forgive me for my slow review.
> >
> > Sure.   No worries.
> >
> > Can you please continue reviewing from the latest v6 ?
> >
> > http://patchwork.ozlabs.org/project/ovn/list/?series=369384
> >
> > I think we can discuss more once you go through all the patches in the
> > series (or until patch 14).
> > I think that would give a better idea of how it is used and we can
> > discuss if it's worth using obj dep mgr.
> >
> > Also I think the patches P1 - P8 make one set and can be considered
> > first.  I submitted all these patches
> > as one series mainly because
> >    -  all the patches can be applied easily
> >    -  the soft freeze was near.
> >
> > The patches can be found here too -
> > https://github.com/numansiddique/ovn/tree/northd_ip_lb_ip_v6
> >
> > Thanks
> > Numan
> >
> > >
> > > While I am still reviewing the patches, I did a performance test with
> > > https://github.com/hzhou8/ovn-test-script for the 200 chassis x 50 lsp
> > > scale of simulated ovn-k8s topology.
> > >
> > > - Before this patch, a recompute takes 1293ms
> > > - After this patch, a recompute takes 1536ms, which is 20% more time.
> > > - I also did the same test with all the rest of the patches of this
series,
> > > which takes 1690ms.
> > >
>
> I did some testing using the OVN dbs of  the ovn-heater density heavy run.
> A triggered forced recompute with the present main of ovn-northd takes
> around 4389ms
> where as with the patch 9 of this series,  it is taking around 4838ms.
>
> I hacked the 'engine_compute_log_timeout_msec' [1] value from 500ms to
> 50ms so that
> inc-eng can log the engine nodes taking more than 50ms in the recompute.
>
> With main
> -------
> 2023-08-18T11:28:20.714Z|00117|inc_proc_eng|INFO|User triggered force
recompute.
> 2023-08-18T11:28:23.189Z|00118|inc_proc_eng|INFO|node: northd,
> recompute (forced) took 2475ms
> 2023-08-18T11:28:25.102Z|00119|inc_proc_eng|INFO|node: lflow,
> recompute (forced) took 1887ms
> 2023-08-18T11:28:25.102Z|00120|timeval|WARN|Unreasonably long 4389ms
> poll interval (4373ms user, 3ms system)
>
>
> And with patch 9
> ---------------------
> 2023-08-18T11:28:10.558Z|00220|inc_proc_eng|INFO|User triggered force
recompute.
> 2023-08-18T11:28:10.883Z|00221|inc_proc_eng|INFO|node: lb_data,
> recompute (forced) took 325ms
> 2023-08-18T11:28:13.163Z|00222|inc_proc_eng|INFO|node: northd,
> recompute (forced) took 2280ms
> 2023-08-18T11:28:13.431Z|00223|inc_proc_eng|INFO|node: sync_to_sb_lb,
> recompute (forced) took 241ms
> 2023-08-18T11:28:15.396Z|00224|inc_proc_eng|INFO|node: lflow,
> recompute (forced) took 1956ms
> 2023-08-18T11:28:15.397Z|00225|timeval|WARN|Unreasonably long 4838ms
> poll interval (4819ms user, 3ms system)
>
>
> As you can see the lflow recompute time is not very significant
> (1887 ms vs 1956 ms)
>
> It is the lb_data engine node which is taking 325ms more.  But if you
> see the lb_data engine code, the lb_data handlers never return false.
> So the lb_data full engine recompute cost can be ignored as it will be
> triggered when engine aborts or when the user triggers a recompute.
>
> I don't think the lflow engine recompute cost due to objdep is
> significant if lflow recomputes are less.
>

My point is that the performance degradation ~20% is caused by just patch
9, because I did the same test on patch 8 v.s. patch 9. (and patch 8 is
similar to main in terms of the recompute time, although I did see a big
percentage increase of time spent for VIF I-P, which I will check later
separately)
I did the same test for v6 (on top of the latest main) between patch 8 and
patch 9. The percentage of the gap is similar:

Before patch9:   1282ms
After patch9:      1495ms

The command I ran was simply creating an empty LR: ovn-nbctl --wait=hv
--print-wait-time lr-add lr-test
Also note that the test is using single thread for ovn-northd.

The lb_data engine node cost shouldn't be a factor because there is no
change of this part in patch 9, right? Maybe it is not objdep that is
causing the performance drop because there are lots of changes in this
patch, but it is hard to tell without profiling/perf.

I didn't spend more time debugging this.  Below is just the result of
stopwatch for reference.
# ovn-appctl -t ovn-northd stopwatch/reset
# ovn-appctl -t ovn-northd inc-engine/recompute
# ovn-appctl -t ovn-northd stopwatch/show
Left side is with patch 9, and the right side is without patch 9:
---
Statistics for 'lflows_to_sb'
 |Statistics for 'lflows_to_sb'
  Total samples: 1                                                     |
 Total samples: 1
  Maximum: 214 msec                                                    |
 Maximum: 197 msec
  Minimum: 214 msec                                                    |
 Minimum: 197 msec
  95th percentile: 0.000000 msec                                       |
 95th percentile: 0.000000 msec
  Short term average: 214.000000 msec                                  |
 Short term average: 197.000000 msec
  Long term average: 214.000000 msec                                   |
 Long term average: 197.000000 msec
Statistics for 'ovnnb_db_run'
 |Statistics for 'ovnnb_db_run'
  Total samples: 1                                                     |
 Total samples: 1
  Maximum: 355 msec                                                    |
 Maximum: 320 msec
  Minimum: 355 msec                                                    |
 Minimum: 320 msec
  95th percentile: 0.000000 msec                                       |
 95th percentile: 0.000000 msec
  Short term average: 355.000000 msec                                  |
 Short term average: 320.000000 msec
  Long term average: 355.000000 msec                                   |
 Long term average: 320.000000 msec
Statistics for 'build_flows_ctx'
|Statistics for 'build_flows_ctx'
  Total samples: 1                                                     |
 Total samples: 1
  Maximum: 346 msec                                                    |
 Maximum: 311 msec
  Minimum: 346 msec                                                    |
 Minimum: 311 msec
  95th percentile: 0.000000 msec                                       |
 95th percentile: 0.000000 msec
  Short term average: 346.000000 msec                                  |
 Short term average: 311.000000 msec
  Long term average: 346.000000 msec                                   |
 Long term average: 311.000000 msec
Statistics for 'ovn-northd-loop'
|Statistics for 'ovn-northd-loop'
  Total samples: 4                                                     |
 Total samples: 4
  Maximum: 1486 msec                                                   |
 Maximum: 1297 msec
  Minimum: 0 msec                                                      |
 Minimum: 0 msec
  95th percentile: 0.000000 msec                                       |
 95th percentile: 0.000000 msec
  Short term average: 743.000000 msec                                  |
 Short term average: 648.750000 msec
  Long term average: 14.860000 msec                                    |
 Long term average: 14.910598 msec
Statistics for 'build_lflows'
 |Statistics for 'build_lflows'
  Total samples: 1                                                     |
 Total samples: 1
  Maximum: 642 msec                                                    |
 Maximum: 588 msec
  Minimum: 642 msec                                                    |
 Minimum: 588 msec
  95th percentile: 0.000000 msec                                       |
 95th percentile: 0.000000 msec
  Short term average: 642.000000 msec                                  |
 Short term average: 588.000000 msec
  Long term average: 642.000000 msec                                   |
 Long term average: 588.000000 msec
Statistics for 'lflows_ports'
 |Statistics for 'lflows_ports'
  Total samples: 1                                                     |
 Total samples: 1
  Maximum: 315 msec                                                    |
 Maximum: 231 msec
  Minimum: 315 msec                                                    |
 Minimum: 231 msec
  95th percentile: 0.000000 msec                                       |
 95th percentile: 0.000000 msec
  Short term average: 315.000000 msec                                  |
 Short term average: 231.000000 msec
  Long term average: 315.000000 msec                                   |
 Long term average: 231.000000 msec

Thanks,
Han

> Thanks
> Numan
>
> > > So the major increase of the recompute time is from this patch. I
didn't
> > > debug what's the exact cause of this increase, but I think it might be
> > > related to the objdep usage.
> > >
> > > > This patch does few more changes which are significant:
> > > >   -  Logical flows are now hashed without the logical
> > > >      datapaths.  If a logical flow is referenced by just one
> > > >      datapath, we don't rehash it.
> > > >
> > > >   -  The synthetic 'hash' column of sbrec_logical_flow now
> > > >      doesn't use the logical datapath too.  This means that
> > > >      when ovn-northd is updated/upgraded and has this commit,
> > > >      all the logical flows with 'logical_datapath' column
> > > >      set will get deleted and re-added causing some disruptions.
> > > >
> > > >   -  With the commit [1] which added I-P support for logical
> > > >      port changes, multiple logical flows with same match 'M'
> > > >      and actions 'A' are generated and stored without the
> > > >      dp groups, which was not the case prior to
> > > >      that patch.
> > > >      One example to generate these lflows is:
> > > >              ovn-nbctl lsp-set-addresses sw0p1 "MAC1 IP1"
> > > >              ovn-nbctl lsp-set-addresses sw1p1 "MAC1 IP1"
> > > >              ovn-nbctl lsp-set-addresses sw2p1 "MAC1 IP1"
> > > >
> > > >      Now with this patch we go back to the earlier way.  i.e
> > > >      one logical flow with logical_dp_groups set.
> > > >
> > > Thanks for taking the effort to achieve this. The approach is indeed
very
> > > smart. I was aware of such scenarios, but it just didn''t seem to
impact
> > > scalability because the number of such lflows should be small, so it
wasn't
> > > considered a priority.
> > >
> > > >   -  With this patch any updates to a logical port which
> > > >      doesn't result in new logical flows will not result in
> > > >      deletion and addition of same logical flows.
> > > >      Eg.
> > > >      ovn-nbctl set logical_switch_port sw0p1 external_ids:foo=bar
> > > >      will be a no-op to the SB logical flow table.
> > > >
> > > Similar to the above case, this doesn't seem to be critical for
> > > scalability, but it is good to see the enhancement.
> > > I will post more feedback after finishing the full review.
> > >
> > > Thanks,
> > > Han
> > >
> > > > [1] - 8bbd67891f68("northd: Incremental processing of VIF additions
in
> > > 'lflow' node.")
> > > >
> > > > Suggested-by: Dumitru Ceara <dceara@redhat.com>
> > > > Signed-off-by: Numan Siddique <numans@ovn.org>
> > > _______________________________________________
> > > dev mailing list
> > > dev@openvswitch.org
> > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev

On Tue, 22 Aug, 2023, 6:11 am Han Zhou, <hzhou@ovn.org> wrote:

> On Fri, Aug 18, 2023 at 4:35 AM Numan Siddique <numans@ovn.org> wrote:
> >
> > On Fri, Aug 18, 2023 at 2:33 PM Numan Siddique <numans@ovn.org> wrote:
> > >
> > > On Fri, Aug 18, 2023 at 11:38 AM Han Zhou <hzhou@ovn.org> wrote:
> > > >
> > > > On Wed, Aug 9, 2023 at 9:38 AM <numans@ovn.org> wrote:
> > > > >
> > > > > From: Numan Siddique <numans@ovn.org>
> > > > >
> > > > Thanks Numan for the enhancement.
> > > >
> > > > > Instead of maintaing the lport to lflow references using
> > > > > 'struct lflow_ref_list', this patch makes use of the existing
> > > > > objdep APIs.  Since objdep_mgr is not thread safe, an instance
> > > > > of objdep_mgr is maintained for each 'ovn_port'.
> > > > >
> > > > > Once we add the thread safe support to objdep APIs we can move
> > > > > the objdep_mgr to 'lflow' engine date.
> > > > >
> > > > > Using objdep APIs does come with a cost in memory.  We now
> > > > > maintain an additional hmap of ovn_lflow's to look up using
> > > > > uuid.  But this will be useful to handle datapath and load
> > > > > balancer changes in the lflow engine node.
> > > > >
> > > >
> > > > I understand that the 'struct lflow_ref_list' may not satisfy the
> needs of
> > > > more complex I-P, but I wonder if objdep_mgr (without modification)
> is the
> > > > solution. It seems to me more complex at least for this patch without
> > > > adding obvious benefits. Since I am still reviewing, maybe I missed
> some
> > > > points which would show the value from the rest of the patches, and
> please
> > > > forgive me for my slow review.
> > >
> > > Sure.   No worries.
> > >
> > > Can you please continue reviewing from the latest v6 ?
> > >
> > > http://patchwork.ozlabs.org/project/ovn/list/?series=369384
> > >
> > > I think we can discuss more once you go through all the patches in the
> > > series (or until patch 14).
> > > I think that would give a better idea of how it is used and we can
> > > discuss if it's worth using obj dep mgr.
> > >
> > > Also I think the patches P1 - P8 make one set and can be considered
> > > first.  I submitted all these patches
> > > as one series mainly because
> > >    -  all the patches can be applied easily
> > >    -  the soft freeze was near.
> > >
> > > The patches can be found here too -
> > > https://github.com/numansiddique/ovn/tree/northd_ip_lb_ip_v6
> > >
> > > Thanks
> > > Numan
> > >
> > > >
> > > > While I am still reviewing the patches, I did a performance test with
> > > > https://github.com/hzhou8/ovn-test-script for the 200 chassis x 50
> lsp
> > > > scale of simulated ovn-k8s topology.
> > > >
> > > > - Before this patch, a recompute takes 1293ms
> > > > - After this patch, a recompute takes 1536ms, which is 20% more time.
> > > > - I also did the same test with all the rest of the patches of this
> series,
> > > > which takes 1690ms.
> > > >
> >
> > I did some testing using the OVN dbs of  the ovn-heater density heavy
> run.
> > A triggered forced recompute with the present main of ovn-northd takes
> > around 4389ms
> > where as with the patch 9 of this series,  it is taking around 4838ms.
> >
> > I hacked the 'engine_compute_log_timeout_msec' [1] value from 500ms to
> > 50ms so that
> > inc-eng can log the engine nodes taking more than 50ms in the recompute.
> >
> > With main
> > -------
> > 2023-08-18T11:28:20.714Z|00117|inc_proc_eng|INFO|User triggered force
> recompute.
> > 2023-08-18T11:28:23.189Z|00118|inc_proc_eng|INFO|node: northd,
> > recompute (forced) took 2475ms
> > 2023-08-18T11:28:25.102Z|00119|inc_proc_eng|INFO|node: lflow,
> > recompute (forced) took 1887ms
> > 2023-08-18T11:28:25.102Z|00120|timeval|WARN|Unreasonably long 4389ms
> > poll interval (4373ms user, 3ms system)
> >
> >
> > And with patch 9
> > ---------------------
> > 2023-08-18T11:28:10.558Z|00220|inc_proc_eng|INFO|User triggered force
> recompute.
> > 2023-08-18T11:28:10.883Z|00221|inc_proc_eng|INFO|node: lb_data,
> > recompute (forced) took 325ms
> > 2023-08-18T11:28:13.163Z|00222|inc_proc_eng|INFO|node: northd,
> > recompute (forced) took 2280ms
> > 2023-08-18T11:28:13.431Z|00223|inc_proc_eng|INFO|node: sync_to_sb_lb,
> > recompute (forced) took 241ms
> > 2023-08-18T11:28:15.396Z|00224|inc_proc_eng|INFO|node: lflow,
> > recompute (forced) took 1956ms
> > 2023-08-18T11:28:15.397Z|00225|timeval|WARN|Unreasonably long 4838ms
> > poll interval (4819ms user, 3ms system)
> >
> >
> > As you can see the lflow recompute time is not very significant
> > (1887 ms vs 1956 ms)
> >
> > It is the lb_data engine node which is taking 325ms more.  But if you
> > see the lb_data engine code, the lb_data handlers never return false.
> > So the lb_data full engine recompute cost can be ignored as it will be
> > triggered when engine aborts or when the user triggers a recompute.
> >
> > I don't think the lflow engine recompute cost due to objdep is
> > significant if lflow recomputes are less.
> >
>
> My point is that the performance degradation ~20% is caused by just patch
> 9, because I did the same test on patch 8 v.s. patch 9. (and patch 8 is
> similar to main in terms of the recompute time, although I did see a big
> percentage increase of time spent for VIF I-P, which I will check later
> separately)
> I did the same test for v6 (on top of the latest main) between patch 8 and
> patch 9. The percentage of the gap is similar:
>
> Before patch9:   1282ms
> After patch9:      1495ms
>
> The command I ran was simply creating an empty LR: ovn-nbctl --wait=hv
> --print-wait-time lr-add lr-test
> Also note that the test is using single thread for ovn-northd.
>
> The lb_data engine node cost shouldn't be a factor because there is no
> change of this part in patch 9, right? Maybe it is not objdep that is
> causing the performance drop because there are lots of changes in this
> patch, but it is hard to tell without profiling/perf.
>
> I didn't spend more time debugging this.  Below is just the result of
> stopwatch for reference.
> # ovn-appctl -t ovn-northd stopwatch/reset
> # ovn-appctl -t ovn-northd inc-engine/recompute
> # ovn-appctl -t ovn-northd stopwatch/show
> Left side is with patch 9, and the right side is without patch 9:
> ---
> Statistics for 'lflows_to_sb'
>  |Statistics for 'lflows_to_sb'
>   Total samples: 1                                                     |
>  Total samples: 1
>   Maximum: 214 msec                                                    |
>  Maximum: 197 msec
>   Minimum: 214 msec                                                    |
>  Minimum: 197 msec
>   95th percentile: 0.000000 msec                                       |
>  95th percentile: 0.000000 msec
>   Short term average: 214.000000 msec                                  |
>  Short term average: 197.000000 msec
>   Long term average: 214.000000 msec                                   |
>  Long term average: 197.000000 msec
> Statistics for 'ovnnb_db_run'
>  |Statistics for 'ovnnb_db_run'
>   Total samples: 1                                                     |
>  Total samples: 1
>   Maximum: 355 msec                                                    |
>  Maximum: 320 msec
>   Minimum: 355 msec                                                    |
>  Minimum: 320 msec
>   95th percentile: 0.000000 msec                                       |
>  95th percentile: 0.000000 msec
>   Short term average: 355.000000 msec                                  |
>  Short term average: 320.000000 msec
>   Long term average: 355.000000 msec                                   |
>  Long term average: 320.000000 msec
> Statistics for 'build_flows_ctx'
> |Statistics for 'build_flows_ctx'
>   Total samples: 1                                                     |
>  Total samples: 1
>   Maximum: 346 msec                                                    |
>  Maximum: 311 msec
>   Minimum: 346 msec                                                    |
>  Minimum: 311 msec
>   95th percentile: 0.000000 msec                                       |
>  95th percentile: 0.000000 msec
>   Short term average: 346.000000 msec                                  |
>  Short term average: 311.000000 msec
>   Long term average: 346.000000 msec                                   |
>  Long term average: 311.000000 msec
> Statistics for 'ovn-northd-loop'
> |Statistics for 'ovn-northd-loop'
>   Total samples: 4                                                     |
>  Total samples: 4
>   Maximum: 1486 msec                                                   |
>  Maximum: 1297 msec
>   Minimum: 0 msec                                                      |
>  Minimum: 0 msec
>   95th percentile: 0.000000 msec                                       |
>  95th percentile: 0.000000 msec
>   Short term average: 743.000000 msec                                  |
>  Short term average: 648.750000 msec
>   Long term average: 14.860000 msec                                    |
>  Long term average: 14.910598 msec
> Statistics for 'build_lflows'
>  |Statistics for 'build_lflows'
>   Total samples: 1                                                     |
>  Total samples: 1
>   Maximum: 642 msec                                                    |
>  Maximum: 588 msec
>   Minimum: 642 msec                                                    |
>  Minimum: 588 msec
>   95th percentile: 0.000000 msec                                       |
>  95th percentile: 0.000000 msec
>   Short term average: 642.000000 msec                                  |
>  Short term average: 588.000000 msec
>   Long term average: 642.000000 msec                                   |
>  Long term average: 588.000000 msec
> Statistics for 'lflows_ports'
>  |Statistics for 'lflows_ports'
>   Total samples: 1                                                     |
>  Total samples: 1
>   Maximum: 315 msec                                                    |
>  Maximum: 231 msec
>   Minimum: 315 msec                                                    |
>  Minimum: 231 msec
>   95th percentile: 0.000000 msec                                       |
>  95th percentile: 0.000000 msec
>   Short term average: 315.000000 msec                                  |
>  Short term average: 231.000000 msec
>   Long term average: 315.000000 msec                                   |
>  Long term average: 231.000000 msec
>


Hi Han,

I'm presently on PTO.  I'll take a look and respond once I'm back.

Meanwhile I kindly request to provide some reviews for the first 8 or 9
patches which are independent of obj dep mgr.

Thanks
Numan


> Thanks,
> Han
>
> > Thanks
> > Numan
> >
> > > > So the major increase of the recompute time is from this patch. I
> didn't
> > > > debug what's the exact cause of this increase, but I think it might
> be
> > > > related to the objdep usage.
> > > >
> > > > > This patch does few more changes which are significant:
> > > > >   -  Logical flows are now hashed without the logical
> > > > >      datapaths.  If a logical flow is referenced by just one
> > > > >      datapath, we don't rehash it.
> > > > >
> > > > >   -  The synthetic 'hash' column of sbrec_logical_flow now
> > > > >      doesn't use the logical datapath too.  This means that
> > > > >      when ovn-northd is updated/upgraded and has this commit,
> > > > >      all the logical flows with 'logical_datapath' column
> > > > >      set will get deleted and re-added causing some disruptions.
> > > > >
> > > > >   -  With the commit [1] which added I-P support for logical
> > > > >      port changes, multiple logical flows with same match 'M'
> > > > >      and actions 'A' are generated and stored without the
> > > > >      dp groups, which was not the case prior to
> > > > >      that patch.
> > > > >      One example to generate these lflows is:
> > > > >              ovn-nbctl lsp-set-addresses sw0p1 "MAC1 IP1"
> > > > >              ovn-nbctl lsp-set-addresses sw1p1 "MAC1 IP1"
> > > > >              ovn-nbctl lsp-set-addresses sw2p1 "MAC1 IP1"
> > > > >
> > > > >      Now with this patch we go back to the earlier way.  i.e
> > > > >      one logical flow with logical_dp_groups set.
> > > > >
> > > > Thanks for taking the effort to achieve this. The approach is indeed
> very
> > > > smart. I was aware of such scenarios, but it just didn''t seem to
> impact
> > > > scalability because the number of such lflows should be small, so it
> wasn't
> > > > considered a priority.
> > > >
> > > > >   -  With this patch any updates to a logical port which
> > > > >      doesn't result in new logical flows will not result in
> > > > >      deletion and addition of same logical flows.
> > > > >      Eg.
> > > > >      ovn-nbctl set logical_switch_port sw0p1 external_ids:foo=bar
> > > > >      will be a no-op to the SB logical flow table.
> > > > >
> > > > Similar to the above case, this doesn't seem to be critical for
> > > > scalability, but it is good to see the enhancement.
> > > > I will post more feedback after finishing the full review.
> > > >
> > > > Thanks,
> > > > Han
> > > >
> > > > > [1] - 8bbd67891f68("northd: Incremental processing of VIF additions
> in
> > > > 'lflow' node.")
> > > > >
> > > > > Suggested-by: Dumitru Ceara <dceara@redhat.com>
> > > > > Signed-off-by: Numan Siddique <numans@ovn.org>
> > > > _______________________________________________
> > > > dev mailing list
> > > > dev@openvswitch.org
> > > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev
> _______________________________________________
> dev mailing list
> dev@openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-dev
>