From patchwork Fri Apr 27 18:57:32 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Soheil Hassas Yeganeh X-Patchwork-Id: 905921 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="IQx15YRx"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 40XjqY1FwZz9s0W for ; Sat, 28 Apr 2018 04:57:45 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1758693AbeD0S5l (ORCPT ); Fri, 27 Apr 2018 14:57:41 -0400 Received: from mail-qt0-f196.google.com ([209.85.216.196]:36673 "EHLO mail-qt0-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1758629AbeD0S5i (ORCPT ); Fri, 27 Apr 2018 14:57:38 -0400 Received: by mail-qt0-f196.google.com with SMTP id q6-v6so3633192qtn.3 for ; Fri, 27 Apr 2018 11:57:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id; bh=+5y37wLk2mzQAov2QOHX8pElS9HLNfhDyzsj/bNhg80=; b=IQx15YRxWqPOuV0p1F+zHr6W951bIEe6cccKUWcce/Gd69tvvjC/gJTuB+ZnS8dA64 xghWVdcDrJ6/3qNjKrWlg7uAtNKmHoYUM85ryNv4hRudPytcuAHoEQg+GFB82VsSZeER WHszkH7b3T2j5FMxykwf56IlQcIw96EAlneSudwSdEVM6p+zWy2l8cOx7kc1nhAi8t58 jFgGNZhRqWnQhGc8TZie91+Ekh7YG7udwQic786DMqOAWdW6zK5TN3qqkcAxzZjtfGky ZyfKTjqGBJJim6X3LMXAkAj3KCwdwUd3StQvv9zpt23Lg7y8C5MIzhw4H/wwdVZQZMJz FXdQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=+5y37wLk2mzQAov2QOHX8pElS9HLNfhDyzsj/bNhg80=; b=PE/AI/f/MqJb4rYoqG/WzgkcoslsqiftmVT+nYidD8YTRnDt5xm2Riyt8oTzv1unhM xDZ/jJrmQIWpCOQYptVVW2PmANgL0LK7CTURo8n8f1hr6LEsxOI+97u9M7dj06G+/g10 zHs42+jk9aLYjlpsyz39kS3x7kQtTNxMMiKr5GJlJ01R3hAFAFf2XEUzE7VNz5dLs8pN C8xWffZ7ymztJPi/xEHrFDUQYTHRLzd3MgHt6VNOk2Nf15f/bTtWlqFvpXQW2WdbwwAq 9h/AaYjHcQ3TDylOtAv2m0DCTdaWuLFxHGkCEobsWEcBvEtW8zlwcWMQe1KuTfswjEvG 0EQA== X-Gm-Message-State: ALQs6tAiTyMMROAWLC1Lx18DNIW3UkPcgMTMBueSqpUvk5DWZ0qV27Yt Gv1nHWcOTueamOfzPkkM5B1Imd/Y X-Google-Smtp-Source: AB8JxZqunH2GXaooaZl7UzUkSFT2sn04eO+9MQMiLnbkrK/So9AAfCfVedcx4WfsqOxl8z7LDMlWZw== X-Received: by 2002:ac8:2329:: with SMTP id a38-v6mr3316725qta.63.1524855457505; Fri, 27 Apr 2018 11:57:37 -0700 (PDT) Received: from z.nyc.corp.google.com ([2620:0:1003:315:9c67:ffa0:44c0:d273]) by smtp.gmail.com with ESMTPSA id r6-v6sm1480327qti.70.2018.04.27.11.57.36 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 27 Apr 2018 11:57:37 -0700 (PDT) From: Soheil Hassas Yeganeh To: davem@davemloft.net, netdev@vger.kernel.org Cc: ycheng@google.com, ncardwell@google.com, edumazet@google.com, willemb@google.com, Soheil Hassas Yeganeh Subject: [PATCH V2 net-next 1/2] tcp: send in-queue bytes in cmsg upon read Date: Fri, 27 Apr 2018 14:57:32 -0400 Message-Id: <20180427185733.36855-1-soheil.kdev@gmail.com> X-Mailer: git-send-email 2.17.0.441.gb46fe60e1d-goog Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Soheil Hassas Yeganeh Applications with many concurrent connections, high variance in receive queue length and tight memory bounds cannot allocate worst-case buffer size to drain sockets. Knowing the size of receive queue length, applications can optimize how they allocate buffers to read from the socket. The number of bytes pending on the socket is directly available through ioctl(FIONREAD/SIOCINQ) and can be approximated using getsockopt(MEMINFO) (rmem_alloc includes skb overheads in addition to application data). But, both of these options add an extra syscall per recvmsg. Moreover, ioctl(FIONREAD/SIOCINQ) takes the socket lock. Add the TCP_INQ socket option to TCP. When this socket option is set, recvmsg() relays the number of bytes available on the socket for reading to the application via the TCP_CM_INQ control message. Calculate the number of bytes after releasing the socket lock to include the processed backlog, if any. To avoid an extra branch in the hot path of recvmsg() for this new control message, move all cmsg processing inside an existing branch for processing receive timestamps. Since the socket lock is not held when calculating the size of receive queue, TCP_INQ is a hint. For example, it can overestimate the queue size by one byte, if FIN is received. With this method, applications can start reading from the socket using a small buffer, and then use larger buffers based on the remaining data when needed. Signed-off-by: Soheil Hassas Yeganeh Signed-off-by: Yuchung Cheng Signed-off-by: Willem de Bruijn Reviewed-by: Eric Dumazet Reviewed-by: Neal Cardwell --- include/linux/tcp.h | 2 +- include/net/tcp.h | 8 ++++++++ include/uapi/linux/tcp.h | 3 +++ net/ipv4/tcp.c | 27 +++++++++++++++++++++++---- 4 files changed, 35 insertions(+), 5 deletions(-) diff --git a/include/linux/tcp.h b/include/linux/tcp.h index 20585d5c4e1c3..807776928cb86 100644 --- a/include/linux/tcp.h +++ b/include/linux/tcp.h @@ -228,7 +228,7 @@ struct tcp_sock { unused:2; u8 nonagle : 4,/* Disable Nagle algorithm? */ thin_lto : 1,/* Use linear timeouts for thin streams */ - unused1 : 1, + recvmsg_inq : 1,/* Indicate # of bytes in queue upon recvmsg */ repair : 1, frto : 1;/* F-RTO (RFC5682) activated in CA_Loss */ u8 repair_queue; diff --git a/include/net/tcp.h b/include/net/tcp.h index 833154e3df173..0986836b5df5b 100644 --- a/include/net/tcp.h +++ b/include/net/tcp.h @@ -1951,6 +1951,14 @@ static inline int tcp_inq(struct sock *sk) return answ; } +static inline int tcp_inq_hint(const struct sock *sk) +{ + const struct tcp_sock *tp = tcp_sk(sk); + + return max_t(int, 0, + READ_ONCE(tp->rcv_nxt) - READ_ONCE(tp->copied_seq)); +} + int tcp_peek_len(struct socket *sock); static inline void tcp_segs_in(struct tcp_sock *tp, const struct sk_buff *skb) diff --git a/include/uapi/linux/tcp.h b/include/uapi/linux/tcp.h index 379b08700a542..d4cdd25a7bd48 100644 --- a/include/uapi/linux/tcp.h +++ b/include/uapi/linux/tcp.h @@ -122,6 +122,9 @@ enum { #define TCP_MD5SIG_EXT 32 /* TCP MD5 Signature with extensions */ #define TCP_FASTOPEN_KEY 33 /* Set the key for Fast Open (cookie) */ #define TCP_FASTOPEN_NO_COOKIE 34 /* Enable TFO without a TFO cookie */ +#define TCP_INQ 35 /* Notify bytes available to read as a cmsg on read */ + +#define TCP_CM_INQ TCP_INQ struct tcp_repair_opt { __u32 opt_code; diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index dfd090ea54ad4..5a7056980f730 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -1910,13 +1910,14 @@ int tcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len, int nonblock, u32 peek_seq; u32 *seq; unsigned long used; - int err; + int err, inq; int target; /* Read at least this many bytes */ long timeo; struct sk_buff *skb, *last; u32 urg_hole = 0; struct scm_timestamping tss; bool has_tss = false; + bool has_cmsg; if (unlikely(flags & MSG_ERRQUEUE)) return inet_recv_error(sk, msg, len, addr_len); @@ -1931,6 +1932,7 @@ int tcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len, int nonblock, if (sk->sk_state == TCP_LISTEN) goto out; + has_cmsg = tp->recvmsg_inq; timeo = sock_rcvtimeo(sk, nonblock); /* Urgent data needs to be handled specially. */ @@ -2117,6 +2119,7 @@ int tcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len, int nonblock, if (TCP_SKB_CB(skb)->has_rxtstamp) { tcp_update_recv_tstamps(skb, &tss); has_tss = true; + has_cmsg = true; } if (TCP_SKB_CB(skb)->tcp_flags & TCPHDR_FIN) goto found_fin_ok; @@ -2136,13 +2139,20 @@ int tcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len, int nonblock, * on connected socket. I was just happy when found this 8) --ANK */ - if (has_tss) - tcp_recv_timestamp(msg, sk, &tss); - /* Clean up data we have read: This will do ACK frames. */ tcp_cleanup_rbuf(sk, copied); release_sock(sk); + + if (has_cmsg) { + if (has_tss) + tcp_recv_timestamp(msg, sk, &tss); + if (tp->recvmsg_inq) { + inq = tcp_inq_hint(sk); + put_cmsg(msg, SOL_TCP, TCP_CM_INQ, sizeof(inq), &inq); + } + } + return copied; out: @@ -3011,6 +3021,12 @@ static int do_tcp_setsockopt(struct sock *sk, int level, tp->notsent_lowat = val; sk->sk_write_space(sk); break; + case TCP_INQ: + if (val > 1 || val < 0) + err = -EINVAL; + else + tp->recvmsg_inq = val; + break; default: err = -ENOPROTOOPT; break; @@ -3436,6 +3452,9 @@ static int do_tcp_getsockopt(struct sock *sk, int level, case TCP_NOTSENT_LOWAT: val = tp->notsent_lowat; break; + case TCP_INQ: + val = tp->recvmsg_inq; + break; case TCP_SAVE_SYN: val = tp->save_syn; break; From patchwork Fri Apr 27 18:57:33 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Soheil Hassas Yeganeh X-Patchwork-Id: 905922 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="dndcfGMC"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 40Xjqg6gQ8z9s08 for ; Sat, 28 Apr 2018 04:57:51 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1758159AbeD0S5s (ORCPT ); Fri, 27 Apr 2018 14:57:48 -0400 Received: from mail-qt0-f193.google.com ([209.85.216.193]:35301 "EHLO mail-qt0-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1758700AbeD0S5o (ORCPT ); Fri, 27 Apr 2018 14:57:44 -0400 Received: by mail-qt0-f193.google.com with SMTP id s2-v6so3631271qti.2 for ; Fri, 27 Apr 2018 11:57:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=9DGHYDagWFWIIZfvabynGiCIviPNAEzeqQNHtv2IlMs=; b=dndcfGMC8sCWpriSWdy0xnTZKugNGw4DrLFmAoo2G9HNhkDy/dAGZHH2qrZLpK4Msm 78Liet3U+e311VrGDSuNnCKvTHI34QK/Ku8XjJx3b+j6W6txe4xMiQtJx/XI6Ca9vv0x Jrr+yLRa0yUtsvDmLDkEEF0cPSIZhEtW/UVvDz/uk8bzNw+r5GOrJ+4jRO/QR960x6XN urGYsG2VN1FjtBjaPiOkRiu9fRhv6A8tRV4MDT4u85MCy1T0yM24GO1d7W+8IvkRQmJD lOHLXQ3dWzyIH+Zjdw5XD+9XyA9HhvRIzcSfK5x5MG/ZfXwwZUdIqTeZuu+O8W4oHfEf rkNA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=9DGHYDagWFWIIZfvabynGiCIviPNAEzeqQNHtv2IlMs=; b=eTkmKzaHVnu0V/wOMwkT6i7mt3hg8a/0bJzIkc2mV20kfhDsghB+MOgO8jvpOJT89L 5yFQx3sD3h+RLxaeAxfQUDlw4mKUTXI81AVZoe9N2RXhuNFUiVL+7nSzEGYzgj0USo0W uqGKZ1kmm/d7+VsO0KgOzf+fOjb5WtVqTKWHT5Kk9rIDr0b1p1a/tNx6nsTiK8mVt+ql a6B5KWL8TWMse4BwIMAnDCXYvwMq984qrRzbqwPGHewXiuLYfkveqGs0aN+Ni8Eu664Q Ww8/jeP/9xSRuaXU79hN2jBCy/qJoXorAmRdrd+TIz4FjQp8P7sZSusQh1NtApq6ltnw K3iA== X-Gm-Message-State: ALQs6tDIn9bIbPRyxBCOGKOYwy+qFMF53SAw8ATCWdn8nmk7Fxk8Uzvr A3eOHrE8tu6ZGS8ftFK0x4o= X-Google-Smtp-Source: AB8JxZqxOxBy5UpUsxUK1vthZb9zEABM7GSD9IlzcAjGaof0Lc7a2VJS116vtycCQO6qfInqi+mV5g== X-Received: by 2002:ac8:836:: with SMTP id u51-v6mr3223252qth.407.1524855463545; Fri, 27 Apr 2018 11:57:43 -0700 (PDT) Received: from z.nyc.corp.google.com ([2620:0:1003:315:9c67:ffa0:44c0:d273]) by smtp.gmail.com with ESMTPSA id r6-v6sm1480327qti.70.2018.04.27.11.57.42 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 27 Apr 2018 11:57:43 -0700 (PDT) From: Soheil Hassas Yeganeh To: davem@davemloft.net, netdev@vger.kernel.org Cc: ycheng@google.com, ncardwell@google.com, edumazet@google.com, willemb@google.com, Soheil Hassas Yeganeh Subject: [PATCH V2 net-next 2/2] selftest: add test for TCP_INQ Date: Fri, 27 Apr 2018 14:57:33 -0400 Message-Id: <20180427185733.36855-2-soheil.kdev@gmail.com> X-Mailer: git-send-email 2.17.0.441.gb46fe60e1d-goog In-Reply-To: <20180427185733.36855-1-soheil.kdev@gmail.com> References: <20180427185733.36855-1-soheil.kdev@gmail.com> Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Soheil Hassas Yeganeh Signed-off-by: Soheil Hassas Yeganeh Signed-off-by: Yuchung Cheng Signed-off-by: Willem de Bruijn Reviewed-by: Eric Dumazet Reviewed-by: Neal Cardwell --- tools/testing/selftests/net/Makefile | 3 +- tools/testing/selftests/net/tcp_inq.c | 189 ++++++++++++++++++++++++++ 2 files changed, 191 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/net/tcp_inq.c diff --git a/tools/testing/selftests/net/Makefile b/tools/testing/selftests/net/Makefile index df9102ec7b7af..0a1821f8dfb18 100644 --- a/tools/testing/selftests/net/Makefile +++ b/tools/testing/selftests/net/Makefile @@ -9,7 +9,7 @@ TEST_PROGS += fib_tests.sh fib-onlink-tests.sh in_netns.sh pmtu.sh udpgso.sh TEST_PROGS += udpgso_bench.sh TEST_GEN_FILES = socket TEST_GEN_FILES += psock_fanout psock_tpacket msg_zerocopy -TEST_GEN_FILES += tcp_mmap +TEST_GEN_FILES += tcp_mmap tcp_inq TEST_GEN_PROGS = reuseport_bpf reuseport_bpf_cpu reuseport_bpf_numa TEST_GEN_PROGS += reuseport_dualstack reuseaddr_conflict TEST_GEN_PROGS += udpgso udpgso_bench_tx udpgso_bench_rx @@ -18,3 +18,4 @@ include ../lib.mk $(OUTPUT)/reuseport_bpf_numa: LDFLAGS += -lnuma $(OUTPUT)/tcp_mmap: LDFLAGS += -lpthread +$(OUTPUT)/tcp_inq: LDFLAGS += -lpthread diff --git a/tools/testing/selftests/net/tcp_inq.c b/tools/testing/selftests/net/tcp_inq.c new file mode 100644 index 0000000000000..3f6a27efbe5cf --- /dev/null +++ b/tools/testing/selftests/net/tcp_inq.c @@ -0,0 +1,189 @@ +/* + * Copyright 2018 Google Inc. + * Author: Soheil Hassas Yeganeh (soheil@google.com) + * + * Simple example on how to use TCP_INQ and TCP_CM_INQ. + * + * License (GPLv2): + * + * This program is free software; you can redistribute it and/or modify it + * under the terms and conditions of the GNU General Public License, + * version 2, as published by the Free Software Foundation. + * + * This program is distributed in the hope it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. * See the GNU General Public License for + * more details. + */ +#define _GNU_SOURCE + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#ifndef TCP_INQ +#define TCP_INQ 35 +#endif + +#ifndef TCP_CM_INQ +#define TCP_CM_INQ TCP_INQ +#endif + +#define BUF_SIZE 8192 +#define CMSG_SIZE 32 + +static int family = AF_INET6; +static socklen_t addr_len = sizeof(struct sockaddr_in6); +static int port = 4974; + +static void setup_loopback_addr(int family, struct sockaddr_storage *sockaddr) +{ + struct sockaddr_in6 *addr6 = (void *) sockaddr; + struct sockaddr_in *addr4 = (void *) sockaddr; + + switch (family) { + case PF_INET: + memset(addr4, 0, sizeof(*addr4)); + addr4->sin_family = AF_INET; + addr4->sin_addr.s_addr = htonl(INADDR_LOOPBACK); + addr4->sin_port = htons(port); + break; + case PF_INET6: + memset(addr6, 0, sizeof(*addr6)); + addr6->sin6_family = AF_INET6; + addr6->sin6_addr = in6addr_loopback; + addr6->sin6_port = htons(port); + break; + default: + error(1, 0, "illegal family"); + } +} + +void *start_server(void *arg) +{ + int server_fd = (int)(unsigned long)arg; + struct sockaddr_in addr; + socklen_t addrlen = sizeof(addr); + char *buf; + int fd; + int r; + + buf = malloc(BUF_SIZE); + + for (;;) { + fd = accept(server_fd, (struct sockaddr *)&addr, &addrlen); + if (fd == -1) { + perror("accept"); + break; + } + do { + r = send(fd, buf, BUF_SIZE, 0); + } while (r < 0 && errno == EINTR); + if (r < 0) + perror("send"); + if (r != BUF_SIZE) + fprintf(stderr, "can only send %d bytes\n", r); + /* TCP_INQ can overestimate in-queue by one byte if we send + * the FIN packet. Sleep for 1 second, so that the client + * likely invoked recvmsg(). + */ + sleep(1); + close(fd); + } + + free(buf); + close(server_fd); + pthread_exit(0); +} + +int main(int argc, char *argv[]) +{ + struct sockaddr_storage listen_addr, addr; + int c, one = 1, inq = -1; + pthread_t server_thread; + char cmsgbuf[CMSG_SIZE]; + struct iovec iov[1]; + struct cmsghdr *cm; + struct msghdr msg; + int server_fd, fd; + char *buf; + + while ((c = getopt(argc, argv, "46p:")) != -1) { + switch (c) { + case '4': + family = PF_INET; + addr_len = sizeof(struct sockaddr_in); + break; + case '6': + family = PF_INET6; + addr_len = sizeof(struct sockaddr_in6); + break; + case 'p': + port = atoi(optarg); + break; + } + } + + server_fd = socket(family, SOCK_STREAM, 0); + if (server_fd < 0) + error(1, errno, "server socket"); + setup_loopback_addr(family, &listen_addr); + if (setsockopt(server_fd, SOL_SOCKET, SO_REUSEADDR, + &one, sizeof(one)) != 0) + error(1, errno, "setsockopt(SO_REUSEADDR)"); + if (bind(server_fd, (const struct sockaddr *)&listen_addr, + addr_len) == -1) + error(1, errno, "bind"); + if (listen(server_fd, 128) == -1) + error(1, errno, "listen"); + if (pthread_create(&server_thread, NULL, start_server, + (void *)(unsigned long)server_fd) != 0) + error(1, errno, "pthread_create"); + + fd = socket(family, SOCK_STREAM, 0); + if (fd < 0) + error(1, errno, "client socket"); + setup_loopback_addr(family, &addr); + if (connect(fd, (const struct sockaddr *)&addr, addr_len) == -1) + error(1, errno, "connect"); + if (setsockopt(fd, SOL_TCP, TCP_INQ, &one, sizeof(one)) != 0) + error(1, errno, "setsockopt(TCP_INQ)"); + + msg.msg_name = NULL; + msg.msg_namelen = 0; + msg.msg_iov = iov; + msg.msg_iovlen = 1; + msg.msg_control = cmsgbuf; + msg.msg_controllen = sizeof(cmsgbuf); + msg.msg_flags = 0; + + buf = malloc(BUF_SIZE); + iov[0].iov_base = buf; + iov[0].iov_len = BUF_SIZE / 2; + + if (recvmsg(fd, &msg, 0) != iov[0].iov_len) + error(1, errno, "recvmsg"); + if (msg.msg_flags & MSG_CTRUNC) + error(1, 0, "control message is truncated"); + + for (cm = CMSG_FIRSTHDR(&msg); cm; cm = CMSG_NXTHDR(&msg, cm)) + if (cm->cmsg_level == SOL_TCP && cm->cmsg_type == TCP_CM_INQ) + inq = *((int *) CMSG_DATA(cm)); + + if (inq != BUF_SIZE - iov[0].iov_len) { + fprintf(stderr, "unexpected inq: %d\n", inq); + exit(1); + } + + printf("PASSED\n"); + free(buf); + close(fd); + return 0; +}