aboutsummaryrefslogtreecommitdiff
path: root/sys/kern/subr_turnstile.c
Commit message (Collapse)AuthorAgeFilesLines
* turnstile: Mention the lock name when panicking due to a sleeping threadMark Johnston2024-08-101-1/+2
| | | | | | | This will hopefully make it a bit easier to track down the cause of such panics. MFC after: 2 weeks
* sys: Remove $FreeBSD$: one-line .c patternWarner Losh2023-08-161-2/+0
| | | | Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/
* ddb: Always terminate DB_SHOW_ALIAS_FLAGS with a semi-colon.John Baldwin2023-07-051-1/+1
| | | | | Reviewed by: kib, markj Differential Revision: https://reviews.freebsd.org/D40818
* ddb: annotate some commands with DB_CMD_MEMSAFEMitchell Horne2022-07-181-1/+1
| | | | | | | | | | This is not completely exhaustive, but covers a large majority of commands in the tree. Reviewed by: markj Sponsored by: Juniper Networks, Inc. Sponsored by: Klara, Inc. Differential Revision: https://reviews.freebsd.org/D35583
* Use atomic loads/stores when updating td->td_stateAlex Richardson2021-02-181-3/+3
| | | | | | | | | | | | | | | KCSAN complains about racy accesses in the locking code. Those races are fine since they are inside a TD_SET_RUNNING() loop that expects the value to be changed by another CPU. Use relaxed atomic stores/loads to indicate that this variable can be written/read by multiple CPUs at the same time. This will also prevent the compiler from doing unexpected re-ordering. Reported by: GENERIC-KCSAN Test Plan: KCSAN no longer complains, kernel still runs fine. Reviewed By: markj, mjg (earlier version) Differential Revision: https://reviews.freebsd.org/D28569
* kern: clean up empty lines in .c and .h filesMateusz Guzik2020-09-011-4/+3
| | | | Notes: svn path=/head/; revision=365222
* ddb(4): show lockchain: Don't dereference LK_KERNPROCConrad Meyer2020-04-021-5/+9
| | | | | | | | | | | | Also, print a little more information for otherwise unhandled inhibited states. Finally, improve the grammar of some prints. Some of the print statements missing verb. Sponsored by: Dell EMC Isilon Notes: svn path=/head/; revision=359581
* Mark more nodes as CTLFLAG_MPSAFE or CTLFLAG_NEEDGIANT (17 of many)Pawel Biernacki2020-02-261-3/+5
| | | | | | | | | | | | | | | | | | | r357614 added CTLFLAG_NEEDGIANT to make it easier to find nodes that are still not MPSAFE (or already are but aren’t properly marked). Use it in preparation for a general review of all nodes. This is non-functional change that adds annotations to SYSCTL_NODE and SYSCTL_PROC nodes using one of the soon-to-be-required flags. Mark all obvious cases as MPSAFE. All entries that haven't been marked as MPSAFE before are by default marked as NEEDGIANT Approved by: kib (mentor, blanket) Commented by: kib, gallatin, melifaro Differential Revision: https://reviews.freebsd.org/D23718 Notes: svn path=/head/; revision=358333
* schedlock 4/4Jeff Roberson2019-12-151-2/+1
| | | | | | | | | | | | | | | | | | | | | Don't hold the scheduler lock while doing context switches. Instead we unlock after selecting the new thread and switch within a spinlock section leaving interrupts and preemption disabled to prevent local concurrency. This means that mi_switch() is entered with the thread locked but returns without. This dramatically simplifies scheduler locking because we will not hold the schedlock while spinning on blocked lock in switch. This change has not been made to 4BSD but in principle it would be more straightforward. Discussed with: markj Reviewed by: kib Tested by: pho Differential Revision: https://reviews.freebsd.org/D22778 Notes: svn path=/head/; revision=355784
* schedlock 3/4Jeff Roberson2019-12-151-14/+36
| | | | | | | | | | | | | Eliminate lock recursion from turnstiles. This was simply used to avoid tracking the top-level turnstile lock. explicitly check for it before picking up and dropping locks. Reviewed by: kib Tested by: pho Differential Revision: https://reviews.freebsd.org/D22746 Notes: svn path=/head/; revision=355782
* schedlock 1/4Jeff Roberson2019-12-151-6/+5
| | | | | | | | | | | | | | | Eliminate recursion from most thread_lock consumers. Return from sched_add() without the thread_lock held. This eliminates unnecessary atomics and lock word loads as well as reducing the hold time for scheduler locks. This will eventually allow for lockless remote adds. Discussed with: kib Reviewed by: jhb Tested by: pho Differential Revision: https://reviews.freebsd.org/D22626 Notes: svn path=/head/; revision=355779
* Fix the turnstile_lock() KPI.Mark Johnston2019-07-241-5/+7
| | | | | | | | | | | | | | | | | | | turnstile_{lock,unlock}() were added for use in epoch. turnstile_lock() returned NULL to indicate that the calling thread had lost a race and the turnstile was no longer associated with the given lock, or the lock owner. However, reader-writer locks may not have a designated owner, in which case turnstile_lock() would return NULL and epoch_block_handler_preempt() would leak spinlocks as a result. Apply a minimal fix: return the lock owner as a separate return value. Reviewed by: kib MFC after: 3 days Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D21048 Notes: svn path=/head/; revision=350310
* subr_turnstile: Extract some common code to a helper.Konstantin Belousov2019-05-161-15/+22
| | | | | | | | | | | | | Code walks the list of contested turnstiles to calculate the priority to unlend. Reviewed by: markj Tested by: pho Sponsored by: The FreeBSD Foundation MFC after: 1 week Notes: svn path=/head/; revision=347694
* Remove an unused argument to turnstile_unpend.Mateusz Guzik2018-06-021-1/+1
| | | | | | | | PR: 228694 Submitted by: Julian Pszczołowski <julian.pszczolowski@gmail.com> Notes: svn path=/head/; revision=334546
* turnstile / sleepqueue: annotate variables only used by debug buildsMatt Macy2018-05-191-2/+2
| | | | Notes: svn path=/head/; revision=333853
* Add simple preempt safe epoch APIMatt Macy2018-05-101-0/+39
| | | | | | | | | | | | | | | | | | Read locking is over used in the kernel to guarantee liveness. This API makes it easy to provide livenes guarantees without atomics. Includes epoch_test kernel module to stress test the API. Documentation will follow initial use case. Test case and improvements to preemption handling in response to discussion with mjg@ Reviewed by: imp@, shurd@ Approved by: sbruno@ Notes: svn path=/head/; revision=333466
* sys/kern: adoption of SPDX licensing ID tags.Pedro F. Giffuni2017-11-271-0/+2
| | | | | | | | | | | | | | | Mainly focus on files that use BSD 2-Clause license, however the tool I was using misidentified many licenses so this was mostly a manual - error prone - task. The Software Package Data Exchange (SPDX) group provides a specification to make it easier for automated tools to detect and summarize well known opensource licenses. We are gradually adopting the specification, noting that the tags are considered only advisory and do not, in any way, superceed or replace the license texts. Notes: svn path=/head/; revision=326271
* Remove unused declaration and update ddb.4Conrad Meyer2017-08-241-3/+0
| | | | | | | | | | | | | A follow-up to r322836. Warnings for the unused declaration were breaking some second tier architectures, but did not show up in Clang on x86. Reported by: markj (ddb.4), emaste (declaration) Sponsored by: Dell EMC Isilon Notes: svn path=/head/; revision=322853
* Merge print_lockchain and print_sleepchainConrad Meyer2017-08-241-71/+17
| | | | | | | | | | | | When debugging a deadlock, it is useful to follow the full chain of locks as far as possible. Reviewed by: jhb Sponsored by: Dell EMC Isilon Differential Revision: https://reviews.freebsd.org/D12115 Notes: svn path=/head/; revision=322836
* ddb(4): Add sleepchains to "show allchains"Conrad Meyer2016-10-221-0/+7
| | | | | | | | | | Reported by: markj Reviewed by: markj Sponsored by: Dell EMC Isilon Differential Revision: https://reviews.freebsd.org/D8320 Notes: svn path=/head/; revision=307780
* Finish r173600. There is no need to test a condition if both casesKonstantin Belousov2016-02-101-6/+3
| | | | | | | | | | | result in the same value. Found by: PVS-Studio Sponsored by: The FreeBSD Foundation MFC after: 1 week Notes: svn path=/head/; revision=295488
* ddb: finish converting boolean values.Pedro F. Giffuni2015-05-211-2/+2
| | | | | | | | | | | | The replacement started at r283088 was necessarily incomplete without replacing boolean_t with bool. This also involved cleaning some type mismatches and ansifying old C function declarations. Pointed out by: bde Discussed with: bde, ian, jhb Notes: svn path=/head/; revision=283248
* dtrace sdt: remove the ugly sname parameter of SDT_PROBE_DEFINEAndriy Gapon2013-11-261-2/+2
| | | | | | | | | | | In its stead use the Solaris / illumos approach of emulating '-' (dash) in probe names with '__' (two consecutive underscores). Reviewed by: markj MFC after: 3 weeks Notes: svn path=/head/; revision=258622
* - For kernel compiled only with KDTRACE_HOOKS and not any lock debuggingAttilio Rao2013-11-251-1/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | option, unbreak the lock tracing release semantic by embedding calls to LOCKSTAT_PROFILE_RELEASE_LOCK() direclty in the inlined version of the releasing functions for mutex, rwlock and sxlock. Failing to do so skips the lockstat_probe_func invokation for unlocking. - As part of the LOCKSTAT support is inlined in mutex operation, for kernel compiled without lock debugging options, potentially every consumer must be compiled including opt_kdtrace.h. Fix this by moving KDTRACE_HOOKS into opt_global.h and remove the dependency by opt_kdtrace.h for all files, as now only KDTRACE_FRAMES is linked there and it is only used as a compile-time stub [0]. [0] immediately shows some new bug as DTRACE-derived support for debug in sfxge is broken and it was never really tested. As it was not including correctly opt_kdtrace.h before it was never enabled so it was kept broken for a while. Fix this by using a protection stub, leaving sfxge driver authors the responsibility for fixing it appropriately [1]. Sponsored by: EMC / Isilon storage division Discussed with: rstone [0] Reported by: rstone [1] Discussed with: philip Notes: svn path=/head/; revision=258541
* Update the comment: we do show the backtrace of misbehaving thread.Pawel Jakub Dawidek2013-02-171-4/+3
| | | | Notes: svn path=/head/; revision=246923
* Improve check coverage about idle threads.Attilio Rao2012-09-121-1/+0
| | | | | | | | | | | | | | | Idle threads are not allowed to acquire any lock but spinlocks. Deny any attempt to do so by panicing at the locking operation when INVARIANTS is on. Then, remove the check on blocking on a turnstile. The check in sleepqueues is left because they are not allowed to use tsleep() either which could happen still. Reviewed by: bde, jhb, kib MFC after: 1 week Notes: svn path=/head/; revision=240424
* Mark the idle threads as non-sleepable and also assert that an idleJohn Baldwin2012-08-221-0/+1
| | | | | | | thread never blocks on a turnstile. Notes: svn path=/head/; revision=239585
* Implement the DTrace sched provider. This implementation aims to beRyan Stone2012-05-151-0/+10
| | | | | | | | | | | | | | | | | | | | | | | | | compatible with the sched provider implemented by Solaris and its open- source derivatives. Full documentation of the sched provider can be found on Oracle's DTrace wiki pages. Note that for compatibility with scripts originally written for Solaris, serveral probes are defined that will never fire. These probes are defined to fire when Solaris-specific features perform certain actions. As these features are not present in FreeBSD, the probes can never fire. Also, I have added a two probes that are not defined in Solaris, lend-pri and load-change. These probes have been added to make it possible to collect schedgraph data with DTrace. Finally, a few probes are defined in Solaris to take a cpuinfo_t * argument. As it was not immediately clear to me how to translate that to FreeBSD, currently those probes are passed NULL in place of a cpuinfo_t *. Sponsored by: Sandvine Incorporated MFC after: 2 weeks Notes: svn path=/head/; revision=235459
* Fix a typo.Davide Italiano2012-04-141-1/+1
| | | | | | | | Approved by: gnn (mentor) MFC after: 2 days Notes: svn path=/head/; revision=234303
* Fix !DDB build after r234190.Marius Strobl2012-04-141-1/+1
| | | | Notes: svn path=/head/; revision=234280
* - Extend the KDB interface to add a per-debugger callback to print aJohn Baldwin2012-04-121-3/+1
| | | | | | | | | | | | | | backtrace for an arbitrary thread (rather than the calling thread). A kdb_backtrace_thread() wrapper function uses the configured debugger if possible, otherwise it falls back to using stack(9) if that is available. - Replace a direct call to db_trace_thread() in propagate_priority() with a call to kdb_backtrace_thread() instead. MFC after: 1 week Notes: svn path=/head/; revision=234190
* Mark all SYSCTL_NODEs static that have no corresponding SYSCTL_DECLs.Ed Schouten2011-11-071-2/+3
| | | | | | | | | The SYSCTL_NODE macro defines a list that stores all child-elements of that node. If there's no SYSCTL_DECL macro anywhere else, there's no reason why it shouldn't be static. Notes: svn path=/head/; revision=227309
* Always assert that the turnstile chain lock is held in turnstile_wait()John Baldwin2011-02-041-2/+1
| | | | | | | | | and remove a duplicate hash lookup. MFC after: 1 week Notes: svn path=/head/; revision=218272
* Introduce the new kernel thread called "deadlock resolver".Attilio Rao2010-01-091-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | While the name is pretentious, a good explanation of its targets is reported in this 17 months old presentation e-mail: http://lists.freebsd.org/pipermail/freebsd-arch/2008-August/008452.html In order to implement it, the sq_type in sleepqueues is mandatory and not only compiled along with INVARIANTS option. Additively, a new sleepqueue function, sleepq_type() is added, returning the type of the sleepqueue linked to a wchan. Three new sysctls are added in order to configure the thread: debug.deadlkres.slptime_threshold debug.deadlkres.blktime_threshold debug.deadlkres.sleepfreq rappresenting the thresholds for sleep and block time that will lead to a deadlock matching (when exceeded), while the sleepfreq rappresents the number of seconds between 2 consecutive thread runnings. In order to enable the deadlock resolver thread recompile your kernel with the option DEADLKRES. Reviewed by: jeff Tested by: pho, Giovanni Trematerra Sponsored by: Nokia Incorporated, Sandvine Incorporated MFC after: 2 weeks Notes: svn path=/head/; revision=201879
* Fix indentation.Ed Schouten2009-12-201-1/+1
| | | | Notes: svn path=/head/; revision=200761
* Make ddb command registration dynamic so modules can extendSam Leffler2008-09-151-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | the command set (only so long as the module is present): o add db_command_register and db_command_unregister to add and remove commands, respectively o replace linker sets with SYSINIT's (and SYSUINIT's) that register commands o expose 3 list heads: db_cmd_table, db_show_table, and db_show_all_table for registering top-level commands, show operands, and show all operands, respectively While here also: o sort command lists o add DB_ALIAS, DB_SHOW_ALIAS, and DB_SHOW_ALL_ALIAS to add aliases for existing commands o add "show all trace" as an alias for "show alltrace" o add "show all locks" as an alias for "show alllocks" Submitted by: Guillaume Ballet <gballet@gmail.com> (original version) Reviewed by: jhb MFC after: 1 month Notes: svn path=/head/; revision=183054
* - Reduce scope of #ifdef's in uma_zcreate() call in init_turnstile0().John Baldwin2008-09-081-3/+4
| | | | | | | | | | | | - Set UMA_ZONE_NOFREE so that the per-turnstile spin locks are type stable to avoid a race where one thread might dereference a lock in a free'd turnstile that was previously used by another thread. Theorized by: tegge (2) MFC after: 1 week Notes: svn path=/head/; revision=182879
* - Make SCHED_STATS more generic by adding a wrapper to create theJeff Roberson2008-04-171-2/+1
| | | | | | | | | | | | | | | | | | | | | variables and sysctl nodes. - In reset walk the children of kern_sched_stats and reset the counters via the oid_arg1 pointer. This allows us to add arbitrary counters to the tree and still reset them properly. - Define a set of switch types to be passed with flags to mi_switch(). These types are named SWT_*. These types correspond to SCHED_STATS counters and are automatically handled in this way. - Make the new SWT_ types more specific than the older switch stats. There are now stats for idle switches, remote idle wakeups, remote preemption ithreads idling, etc. - Add switch statistics for ULE's pickcpu algorithm. These stats include how much migration there is, how often affinity was successful, how often threads were migrated to the local cpu on wakeup, etc. Sponsored by: Nokia Notes: svn path=/head/; revision=178272
* - Add THREAD_LOCKPTR_ASSERT() to assert that the thread's lock points atJeff Roberson2008-02-071-7/+7
| | | | | | | | | | | | | the provided lock or &blocked_lock. The thread may be temporarily assigned to the blocked_lock by the scheduler so a direct comparison can not always be made. - Use THREAD_LOCKPTR_ASSERT() in the primary consumers of the scheduling interfaces. The schedulers themselves still use more explicit asserts. Sponsored by: Nokia Notes: svn path=/head/; revision=176078
* Adaptive spinning in write path with readers and writer starvation avoidance.Jeff Roberson2008-02-061-8/+3
| | | | | | | | | | | | | | | | | | | - Move recursion checking into rwlock inlines to free a bit for use with adaptive spinners. - Clear the RW_LOCK_WRITE_SPINNERS flag whenever the lock state changes causing write spinners to restart their loop. - Write spinners are limited by a count while readers hold the lock as there is no way to know for certain whether readers are running still. - In the read path block if there are write waiters or spinners to avoid starving writers. Use a new per-thread count, td_rw_rlocks, to skip starvation avoidance if it might cause a deadlock. - Remove or change invalid assertions in turnstiles. Reviewed by: attilio (developed parts of the patch as well) Sponsored by: Nokia Notes: svn path=/head/; revision=176017
* generally we are interested in what thread did something asJulian Elischer2007-11-141-4/+4
| | | | | | | | | opposed to what process. Since threads by default have teh name of the process unless over-written with more useful information, just print the thread name instead. Notes: svn path=/head/; revision=173600
* - Include opt_sched.h for SCHED_STATS.Jeff Roberson2007-06-121-0/+1
| | | | Notes: svn path=/head/; revision=170640
* Commit 3/14 of sched_lock decomposition.Jeff Roberson2007-06-041-135/+149
| | | | | | | | | | | | | | | | | | | | - Add a per-turnstile spinlock to solve potential priority propagation deadlocks that are possible with thread_lock(). - The turnstile lock order is defined as the exact opposite of the lock order used with the sleep locks they represent. This allows us to walk in reverse order in priority_propagate and this is the only place we wish to multiply acquire turnstile locks. - Use the turnstile_chain lock to protect assigning mutexes to turnstiles. - Change the turnstile interface to pass back turnstile pointers to the consumers. This allows us to reduce some locking and makes it easier to cancel turnstile assignment while the turnstile chain lock is held. Tested by: kris, current@ Tested on: i386, amd64, ULE, 4BSD, libthr, libkse, PREEMPTION, etc. Discussed with: kris, attilio, kmacy, jhb, julian, bde (small parts each) Notes: svn path=/head/; revision=170295
* - Convert turnstiles and sleepqueus to use UMA. This provides a modestJeff Roberson2007-05-181-21/+54
| | | | | | | | | | | | | speedup and will be more useful after each gains a spinlock in the impending thread_lock() commit. - Move initialization and asserts into init/fini routines. fini routines are only needed in the INVARIANTS case for now. Submitted by: Attilio Rao <attilio@FreeBSD.org> Tested by: kris, jeff Notes: svn path=/head/; revision=169666
* - Remove setrunqueue and replace it with direct calls to sched_add().Jeff Roberson2007-01-231-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | setrunqueue() was mostly empty. The few asserts and thread state setting were moved to the individual schedulers. sched_add() was chosen to displace it for naming consistency reasons. - Remove adjustrunqueue, it was 4 lines of code that was ifdef'd to be different on all three schedulers where it was only called in one place each. - Remove the long ifdef'd out remrunqueue code. - Remove the now redundant ts_state. Inspect the thread state directly. - Don't set TSF_* flags from kern_switch.c, we were only doing this to support a feature in one scheduler. - Change sched_choose() to return a thread rather than a td_sched. Also, rely on the schedulers to return the idlethread. This simplifies the logic in choosethread(). Aside from the run queue links kern_switch.c mostly does not care about the contents of td_sched. Discussed with: julian - Move the idle thread loop into the per scheduler area. ULE wants to do something different from the other schedulers. Suggested by: jhb Tested on: x86/amd64 sched_{4BSD, ULE, CORE}. Notes: svn path=/head/; revision=166188
* Use FOREACH_PROC_IN_SYSTEM instead of using its unrolled form.Xin LI2007-01-171-1/+1
| | | | Notes: svn path=/head/; revision=166073
* Wrap propagate_priority() in a critical section to prevent unwantedJohn Baldwin2007-01-111-0/+4
| | | | | | | | | | | | preemptions when adjusting the priority of a thread that is on a run queue. This was only observed when FULL_PREEMPTION was enabled. Reported by: kris Diagnosed by: ups MFC after: 1 week Notes: svn path=/head/; revision=165946
* Add a new 'show sleepchain' ddb command similar to 'show lockchain' exceptJohn Baldwin2006-08-151-0/+67
| | | | | | | | | | | | | that it operates on lockmgr and sx locks. This can be useful for tracking down vnode deadlocks in VFS for example. Note that this command is a bit more fragile than 'show lockchain' as we have to poke around at the wait channel of a thread to see if it points to either a struct lock or a condition variable inside of a struct sx. If td_wchan points to something unmapped, then this command will terminate early due to a fault, but no harm will be done. Notes: svn path=/head/; revision=161337
* Rename 'show lockchain' to 'show locktree' and 'show threadchain' toJohn Baldwin2006-08-151-5/+9
| | | | | | | | | | | 'show lockchain'. The churn is because I'm about to add a new 'show sleepchain' similar to 'show lockchain' for sleep locks (lockmgr and sx) and 'show threadchain' was a bit ambiguous as both commands show a chain of thread dependencies, 'lockchain' is for non-sleepable locks (mtx and rw) and 'sleepchain' is for sleepable locks. Notes: svn path=/head/; revision=161324
* Honor db_pager_quit in 'show threadchain', 'show allchains', andJohn Baldwin2006-07-121-1/+7
| | | | | | | | 'show lockchain'. This is especially helpful for the first 2 as a threadchain could get stuck in an infinite loop during a mutex deadlock. Notes: svn path=/head/; revision=160313
generated by cgit v1.2.3 (git 2.25.1) at 2024-10-23 12:19:47 +0000