aboutsummaryrefslogtreecommitdiff
path: root/sys/netinet/tcp_hpts.c
Commit message (Collapse)AuthorAgeFilesLines
* tcp hpts: initialize variableMichael Tuexen2024-08-011-1/+1
| | | | | | | | | | | | Ensure that tv.tv_sec is zero in all code paths. Reported by: Coverity Scan CID: 1527724 Reviewed by: rscheff Sponsored by: Netflix, Inc. Differential Revision: https://reviews.freebsd.org/D44584 (cherry picked from commit aaaa01c0c858fd703194c6cbd515dd514574381f)
* tcp hpts: improve consistencyMichael Tuexen2024-08-011-1/+2
| | | | | | | | | | | | | | The target_slot argument of max_slots_available() can be NULL. Therefore, check for this in all places. Right now, all callers provide non-NULL pointer. Reported by: Coverity Scan CID: 1527732 Reviewed by: rrs Sponsored by: Netflix, Inc. Differential Revision: https://reviews.freebsd.org/D44527 (cherry picked from commit b600644fdd6cefb1b90d76fdd5aa595946611a7d)
* tcp_hpts: Fix a typo of a function name in a commentGordon Bergling2024-01-231-1/+1
| | | | | | - s/tcp_ouput/tcp_output/ (cherry picked from commit ef0ac0a1ad6750291b881203030384b7f7241efb)
* sys: Remove $FreeBSD$: one-line .c patternWarner Losh2023-08-231-2/+0
| | | | | | | Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/ Similar commit in current: (cherry picked from commit 685dc743dc3b)
* tcp_hpts: Fix a typo in a source code commentGordon Bergling2023-01-141-1/+1
| | | | | | - s/subract/subtract/ (cherry picked from commit d68f15420528e11f69a7bda365608581bfc34e64)
* tcp_hpts: Correct some typos in source code commentsGordon Bergling2022-09-071-2/+2
| | | | | | | - s/occured/occurred/ - s/the the/the/ (cherry picked from commit d07a5018766e11d7f9d7f0c1aa5e0a9baa5ca6d6)
* Fix unused variable warnings in tcp_hpts.cDimitry Andric2022-08-211-4/+0
| | | | | | | | | | | | | | | | | | | | With clang 15, the following -Werror warning is produced: sys/netinet/tcp_hpts.c:1114:10: error: variable 'paced_cnt' set but not used [-Werror,-Wunused-but-set-variable] int32_t paced_cnt = 0; ^ sys/netinet/tcp_hpts.c:1112:11: error: variable 'total_slots_processed' set but not used [-Werror,-Wunused-but-set-variable] uint64_t total_slots_processed = 0; ^ The 'paced_cnt' variable was in tcp_hpts.c when it was first added, and the 'total_slots_processed' variable was added in d7955cc0ffdf9, but both appear to have been debugging aids that have never been used, so remove them. MFC after: 3 days (cherry picked from commit b33bfe6e157429ad764e9a883a5e3a194a6d1f9f)
* Adjust function definition in tcp_hpts.c to avoid clang 15 warningDimitry Andric2022-08-211-1/+1
| | | | | | | | | | | | | | | | | With clang 15, the following -Werror warning is produced: sys/netinet/tcp_hpts.c:1594:23: error: a function declaration without a prototype is deprecated in all versions of C [-Werror,-Wstrict-prototypes] tcp_choose_hpts_to_run() ^ void This is because tcp_choose_hpts_to_run() is declared with a (void) argument list, but defined with an empty argument list. Make the definition match the declaration. MFC after: 3 days (cherry picked from commit db6b32867d6ed996f8a9aad41ca6ffbaa2088e59)
* tcp_htps: Fix a typo in a source code commentGordon Bergling2022-04-141-1/+1
| | | | | | - s/postion/position/ (cherry picked from commit 1f2aaef29a17534ae397227c4d54657eef39dd74)
* netinet: simplify RSS ifdef statementsFranco Fichtner2022-03-031-3/+2
| | | | | | | | Approved by: transport (rrs) MFC after: 3 days Differential Revision: https://reviews.freebsd.org/D31583 (cherry picked from commit 47ded797ce9620f055f005e9dccfcd03613b1804)
* tcp: Fix 32 bit platform breakageRandall Stewart2021-07-131-2/+2
| | | | | | | | | | | | This fixes the incorrect use of a sysctl add to u64. It was for a useconds time, but on 32 bit platforms its not a u64. Instead use the long directive. Reviewed by: tuexen Sponsored by: Netflix Inc. Differential Revision: https://reviews.freebsd.org/D31107 (cherry picked from commit 7312e4e5cfc8e48597acf17f4faa8159f0b5fa06)
* tcp: HPTS performance enhancementsRandall Stewart2021-07-131-513/+796
| | | | | | | | | | | | | | | HPTS drives both rack and bbr, and yet there have been many complaints about performance. This bit of work restructures hpts to help reduce CPU overhead. It does this by now instead of relying on the timer/callout to drive it instead use user return from a system call as well as lro flushes to drive hpts. The timer becomes a backstop that dynamically adjusts based on how "late" we are. Reviewed by: tuexen, glebius Sponsored by: Netflix Inc. Differential Revision: https://reviews.freebsd.org/D31083 (cherry picked from commit d7955cc0ffdf9fb58013245a6f181c757574ea0a)
* net: clean up empty lines in .c and .h filesMateusz Guzik2020-09-011-3/+0
| | | | Notes: svn path=/head/; revision=365071
* tcp_hpts: make RSS kernel compile again.Bjoern A. Zeeb2020-03-031-4/+11
| | | | | | | | | | | | Add proper #includes, and #ifdefs and some style fixes to make RSS kernels compile again. There are still possible issues with uin16_t vs. uint_t cpuid which I am not going near. Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D23726 Notes: svn path=/head/; revision=358577
* Mark more nodes as CTLFLAG_MPSAFE or CTLFLAG_NEEDGIANT (17 of many)Pawel Biernacki2020-02-261-3/+4
| | | | | | | | | | | | | | | | | | | r357614 added CTLFLAG_NEEDGIANT to make it easier to find nodes that are still not MPSAFE (or already are but aren’t properly marked). Use it in preparation for a general review of all nodes. This is non-functional change that adds annotations to SYSCTL_NODE and SYSCTL_PROC nodes using one of the soon-to-be-required flags. Mark all obvious cases as MPSAFE. All entries that haven't been marked as MPSAFE before are by default marked as NEEDGIANT Approved by: kib (mentor, blanket) Commented by: kib, gallatin, melifaro Differential Revision: https://reviews.freebsd.org/D23718 Notes: svn path=/head/; revision=358333
* Whitespace, remove from three files trailing whiteRandall Stewart2020-02-121-60/+60
| | | | | | | | | space (leftover presents from emacs). Sponsored by: Netflix Inc. Notes: svn path=/head/; revision=357817
* In TCP HPTS enter the epoch in tcp_hpts_thread() and assert it inGleb Smirnoff2019-11-071-21/+7
| | | | | | | the leaf functions. Notes: svn path=/head/; revision=354487
* This commit updates rack to what is basically being used at NF asRandall Stewart2019-07-101-632/+705
| | | | | | | | | | | | | | well as sets in some of the groundwork for committing BBR. The hpts system is updated as well as some other needed utilities for the entrance of BBR. This is actually part 1 of 3 more needed commits which will finally complete with BBRv1 being added as a new tcp stack. Sponsored by: Netflix Inc. Differential Revision: https://reviews.freebsd.org/D20834 Notes: svn path=/head/; revision=349893
* Bind TCP HPTS (pacer) threads to NUMA domainsAndrew Gallatin2019-05-101-10/+60
| | | | | | | | | | | | | | | | | | | Bind the TCP pacer threads to NUMA domains and build per-domain pacer-thread lookup tables. These tables allow us to use the inpcb's NUMA domain information to match an inpcb with a pacer thread on the same domain. The motivation for this is to keep the TCP connection local to a NUMA domain as much as possible. Thanks to jhb for pre-reviewing an earlier version of the patch. Reviewed by: rrs Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20134 Notes: svn path=/head/; revision=347430
* Regularize the Netflix copyrightWarner Losh2019-02-041-1/+1
| | | | | | | | | | | | | | | Use recent best practices for Copyright form at the top of the license: 1. Remove all the All Rights Reserved clauses on our stuff. Where we piggybacked others, use a separate line to make things clear. 2. Use "Netflix, Inc." everywhere. 3. Use a single line for the copyright for grep friendliness. 4. Use date ranges in all places for our stuff. Approved by: Netflix Legal (who gave me the form), adrian@ (pmc files) Notes: svn path=/head/; revision=343755
* Add INP_INFO_WUNLOCK_ASSERT() macro and use it instead ofAndrey V. Elsukov2018-10-011-2/+2
| | | | | | | | | | | | | | INP_INFO_UNLOCK_ASSERT() in TCP-related code. For encapsulated traffic it is possible, that the code is running in net_epoch_preempt section, and INP_INFO_UNLOCK_ASSERT() is very strict assertion for such case. PR: 231428 Reviewed by: mmacy, tuexen Approved by: re (kib) Differential Revision: https://reviews.freebsd.org/D17335 Notes: svn path=/head/; revision=339039
* Make tcp_hpts.c compile a LINT kernel with options RSS and PCBGROUPS added byBjoern A. Zeeb2018-09-061-1/+3
| | | | | | | | | | | | adding the missing include files and changing a the type of cpuid which would otherwise cause a false comparison with NETISR_CPUID_NONE. Reviewed by: rrs Approved by: re (marius) Differential Revision: https://reviews.freebsd.org/D16891 Notes: svn path=/head/; revision=338498
* Remove unneeded ipsec-related includes.Andrey V. Elsukov2018-08-101-5/+0
| | | | | | | | Reviewed by: rrs Differential Revision: https://reviews.freebsd.org/D16637 Notes: svn path=/head/; revision=337568
* epoch(9): allow preemptible epochs to composeMatt Macy2018-07-041-13/+16
| | | | | | | | | | | | | | | | | | | | | | | - Add tracker argument to preemptible epochs - Inline epoch read path in kernel and tied modules - Change in_epoch to take an epoch as argument - Simplify tfb_tcp_do_segment to not take a ti_locked argument, there's no longer any benefit to dropping the pcbinfo lock and trying to do so just adds an error prone branchfest to these functions - Remove cases of same function recursion on the epoch as recursing is no longer free. - Remove the the TAILQ_ENTRY and epoch_section from struct thread as the tracker field is now stack or heap allocated as appropriate. Tested by: pho and Limelight Networks Reviewed by: kbowling at llnw dot com Sponsored by: Limelight Networks Differential Revision: https://reviews.freebsd.org/D16066 Notes: svn path=/head/; revision=335924
* Move the tp set back to where it was beforeRandall Stewart2018-06-191-1/+1
| | | | | | | | | | | | we started playing with the VNET sets. This way we have verified the INP settings before we go to the trouble of de-referencing it. Reviewed by: and suggested by lstewart Sponsored by: Netflix Inc. Notes: svn path=/head/; revision=335361
* convert inpcbinfo hash and info rwlocks to epoch + mutexMatt Macy2018-06-191-63/+2
| | | | | | | | | | | | | | | | | | | | | | | | | - Convert inpcbinfo info & hash locks to epoch for read and mutex for write - Garbage collect code that handled INP_INFO_TRY_RLOCK failures as INP_INFO_RLOCK which can no longer fail When running 64 netperfs sending minimal sized packets on a 2x8x2 reduces unhalted core cycles samples in rwlock rlock/runlock in udp_send from 51% to 3%. Overall packet throughput rate limited by CPU affinity and NIC driver design choices. On the receiver unhalted core cycles samples in in_pcblookup_hash went from 13% to to 1.6% Tested by LLNW and pho@ Reviewed by: jtl Sponsored by: Limelight Networks Differential Revision: https://reviews.freebsd.org/D15686 Notes: svn path=/head/; revision=335356
* Move to using the inp->vnet pointer has suggested by lstewart.Randall Stewart2018-06-181-2/+2
| | | | | | | | | | This is far better since the hpts system is using the inp as its basis anyway. Unfortunately his comments came late. Sponsored by: Netflix Inc. Notes: svn path=/head/; revision=335317
* TCPOUTFLAGS no longer exists since r334843.Gleb Smirnoff2018-06-141-1/+0
| | | | Notes: svn path=/head/; revision=335180
* This fixes missing VNET sets in the hpts system. BasicallyRandall Stewart2018-06-121-3/+4
| | | | | | | | | | | | without this and running vnets with a TCP stack that uses some of the features is a recipe for panic (without this commit). Reported by: Larry Rosenman Sponsored by: Netflix Inc. Differential Revision: https://reviews.freebsd.org/D15757 Notes: svn path=/head/; revision=335022
* Change RACK dependency on TCPHPTS from a build-time dependency to a load-Jonathan T. Looney2018-06-111-0/+1
| | | | | | | | | | | | | | | time dependency. At present, RACK requires the TCPHPTS option to run. However, because modules can be moved from machine to machine, this dependency is really best assessed at load time rather than at build time. Reviewed by: rrs Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D15756 Notes: svn path=/head/; revision=334949
* hpts: remove redundant decl breaking gcc buildMatt Macy2018-06-081-1/+0
| | | | Notes: svn path=/head/; revision=334842
* Minor style nitsWarner Losh2018-05-091-2/+1
| | | | | | | | | | | | | Use full copyright year. Remove 'All Rights Reserved' from new file (rights holder OK'd) Minor #ifdef motion and #endif tagging Remove __FBSDID macro from comments Sponsored by: Netflix OK'd by: rrs@ Notes: svn path=/head/; revision=333409
* This commit brings in the TCP high precision timer system (tcp_hpts).Randall Stewart2018-04-191-0/+1964
It is the forerunner/foundational work of bringing in both Rack and BBR which use hpts for pacing out packets. The feature is optional and requires the TCPHPTS option to be enabled before the feature will be active. TCP modules that use it must assure that the base component is compile in the kernel in which they are loaded. MFC after: Never Sponsored by: Netflix Inc. Differential Revision: https://reviews.freebsd.org/D15020 Notes: svn path=/head/; revision=332770
generated by cgit v1.2.3 (git 2.25.1) at 2025-02-28 04:06:21 +0000