-----BEGIN PGP SIGNATURE-----
iQIcBAABAgAGBQJYnZNdAAoJEE44bZycYXAvJv0P/jpPc+jKb+D0FVUOiYDkY5Rw
jxsZ3oeruTIeSFAAIzusVMLm9moBJA6DThuTHU5Kt68mRaKB2lgmqwkkvQAPTSYh
tnDQwlrF7dOSVmPczJFHalpaLpRdXdQP9r8y+38PibaFZPssKdnZr3BBfdOdi5DT
lj029AGKfG7co6Hb/iAhsxuAFfPmvGHY4QNwJ2FRbU1m6MDtmCTbXzF0fc6X5AW1
qrtaWwPulJtZ/5MPk7aFyNpuCpNvIaTEqNaQsZbuz3bHfzDQVLerWze98vgHC0QM
2YOTP6TnEiHhxHGMb9SywUgSV1ylx0X542YDfxmcfyxBWRr0khlxQh1gpX+waqE3
pqdSlvN7AFzifw6kubbG2/XjkNvFtJcDTgrL3qco4utIezSijXmoOsDpKNnJuzk/
kSD5WYd+Q1CSHOkqZX29QPw1Dl/7Ftm7GPfxu7Pis1OBuPByqtRkEfmn9DpiKSs5
Aja0ljZYiQ3jy3fH+WlEzo6PVSxx0ZxKg0fOShlpgjj8KjMUdGfl9cB1OZxyWnNH
UiQ9iIWd3tJci7WbsBOfawsQpq3EIJxZKjyUmLYpBht5/YenYxOBDCr/CLJDQBGI
IQUPAs/E1JGDxGTUY3AmsaMVrcX2yOfhLzjrsVJGqSdote0um+2PdTLZHE4MMiz2
Dh6CbUVYWS1KNgmQ8T8L
=k5mW
-----END PGP SIGNATURE-----
gpgsig -----BEGIN PGP SIGNATURE-----
iQIzBAABCAAdFiEEJDfLduVEy2qz2d/TmXOSYMtstxYFAlpqeiwACgkQmXOSYMts
txaBAQ/+KqZh90YZI+gRHGdczbo3XnlryHMdpp+DTIFtN3zU+2LM352oP+haoJfr
YhNsixcMhW5TX0is5fg4SkIc0B3ooGKZLVKOPIRw+1NLBAVG5yVuYxW7I1faJgk6
F37+4rvq7KAOPCNMjAEXRt7GqZ4WZjgvgKy+u5wzKh3k5kUylqDwlP2qdgx2L5Rc
IxyxgOuaVGV6dZTyAyRlRMild5Tlz+SMY4pWoMe0sulDDXhd5/5PnGNVIgh+XqB6
m0AGkIIzPVe+wmg6n1iYs93dQO0Jmu6DL47Zv4f3ASZNL/XVSLvU9ie63FyWGZXG
e52qAPtztXInEOo15vPQSAAq7McZHDTzhHhsU/ZtkBT+LeSUU+rsxXddJ2EO5UgC
O3cVm11x1FWMzbBtFNFtkqeri2Y2OxvU4O81mfNP1oOUQBTMeSHTzQ8psbCdXeEr
ktSOtI+nakPmDE3aq4YSaz7BwSgt2tU/vZehkrTxtAQJxt0b88r2xFfThy5WScT1
v6muoqxlprjjvFld7v99P8cXxJq4QrxKUxXtEBTdB79Q5xtCC29OAcTelpPFDCED
/KpgZflubzH/Z872AW9Ru8OL9PYty6hBNDOP4aHLSFWfCu3KQxL6BMEeqi5qBjBX
mJ8JT0dCQYP6xONIWq6a3fICroNMazhNFxdpPSfsQFRhujhjGPg=
=zhKv
-----END PGP SIGNATURE-----
Merge 3.10.105 into android-msm-bullhead-3.10-oreo-m5
Changes in 3.10.105: (315 commits)
sched/core: Fix a race between try_to_wake_up() and a woken up task
sched/core: Fix an SMP ordering race in try_to_wake_up() vs. schedule()
crypto: algif_skcipher - Require setkey before accept(2)
crypto: af_alg - Disallow bind/setkey/... after accept(2)
crypto: af_alg - Add nokey compatibility path
crypto: algif_skcipher - Add nokey compatibility path
crypto: hash - Add crypto_ahash_has_setkey
crypto: shash - Fix has_key setting
crypto: algif_hash - Require setkey before accept(2)
crypto: skcipher - Add crypto_skcipher_has_setkey
crypto: algif_skcipher - Add key check exception for cipher_null
crypto: af_alg - Allow af_af_alg_release_parent to be called on nokey path
crypto: algif_hash - Remove custom release parent function
crypto: algif_skcipher - Remove custom release parent function
crypto: af_alg - Forbid bind(2) when nokey child sockets are present
crypto: algif_hash - Fix race condition in hash_check_key
crypto: algif_skcipher - Fix race condition in skcipher_check_key
crypto: algif_skcipher - Load TX SG list after waiting
crypto: cryptd - initialize child shash_desc on import
crypto: skcipher - Fix blkcipher walk OOM crash
crypto: gcm - Fix IV buffer size in crypto_gcm_setkey
MIPS: KVM: Fix unused variable build warning
KVM: MIPS: Precalculate MMIO load resume PC
KVM: MIPS: Drop other CPU ASIDs on guest MMU changes
KVM: nVMX: postpone VMCS changes on MSR_IA32_APICBASE write
KVM: MIPS: Make ERET handle ERL before EXL
KVM: x86: fix wbinvd_dirty_mask use-after-free
KVM: x86: fix missed SRCU usage in kvm_lapic_set_vapic_addr
KVM: Disable irq while unregistering user notifier
PM / devfreq: Fix incorrect type issue.
ppp: defer netns reference release for ppp channel
x86/mm/xen: Suppress hugetlbfs in PV guests
xen: Add RING_COPY_REQUEST()
xen-netback: don't use last request to determine minimum Tx credit
xen-netback: use RING_COPY_REQUEST() throughout
xen-blkback: only read request operation from shared ring once
xen/pciback: Save xen_pci_op commands before processing it
xen/pciback: Save the number of MSI-X entries to be copied later.
xen/pciback: Return error on XEN_PCI_OP_enable_msi when device has MSI or MSI-X enabled
xen/pciback: Return error on XEN_PCI_OP_enable_msix when device has MSI or MSI-X enabled
xen/pciback: Do not install an IRQ handler for MSI interrupts.
xen/pciback: For XEN_PCI_OP_disable_msi[|x] only disable if device has MSI(X) enabled.
xen/pciback: Don't allow MSI-X ops if PCI_COMMAND_MEMORY is not set.
xen-pciback: Add name prefix to global 'permissive' variable
x86/xen: fix upper bound of pmd loop in xen_cleanhighmap()
x86/traps: Ignore high word of regs->cs in early_idt_handler_common
x86/mm: Disable preemption during CR3 read+write
x86/apic: Do not init irq remapping if ioapic is disabled
x86/mm/pat, /dev/mem: Remove superfluous error message
x86/paravirt: Do not trace _paravirt_ident_*() functions
x86/build: Build compressed x86 kernels as PIE
x86/um: reuse asm-generic/barrier.h
iommu/amd: Update Alias-DTE in update_device_table()
iommu/amd: Free domain id when free a domain of struct dma_ops_domain
ARM: 8616/1: dt: Respect property size when parsing CPUs
ARM: 8618/1: decompressor: reset ttbcr fields to use TTBR0 on ARMv7
ARM: sa1100: clear reset status prior to reboot
ARM: sa1111: fix pcmcia suspend/resume
arm64: avoid returning from bad_mode
arm64: Define AT_VECTOR_SIZE_ARCH for ARCH_DLINFO
arm64: spinlocks: implement smp_mb__before_spinlock() as smp_mb()
arm64: debug: avoid resetting stepping state machine when TIF_SINGLESTEP
MIPS: Malta: Fix IOCU disable switch read for MIPS64
MIPS: ptrace: Fix regs_return_value for kernel context
powerpc/mm: Don't alias user region to other regions below PAGE_OFFSET
powerpc/vdso64: Use double word compare on pointers
powerpc/powernv: Use CPU-endian PEST in pnv_pci_dump_p7ioc_diag_data()
powerpc/64: Fix incorrect return value from __copy_tofrom_user
powerpc/nvram: Fix an incorrect partition merge
avr32: fix copy_from_user()
avr32: fix 'undefined reference to `___copy_from_user'
avr32: off by one in at32_init_pio()
s390/dasd: fix hanging device after clear subchannel
parisc: Ensure consistent state when switching to kernel stack at syscall entry
microblaze: fix __get_user()
microblaze: fix copy_from_user()
mn10300: failing __get_user() and get_user() should zero
m32r: fix __get_user()
sh64: failing __get_user() should zero
score: fix __get_user/get_user
s390: get_user() should zero on failure
ARC: uaccess: get_user to zero out dest in cause of fault
asm-generic: make get_user() clear the destination on errors
frv: fix clear_user()
cris: buggered copy_from_user/copy_to_user/clear_user
blackfin: fix copy_from_user()
score: fix copy_from_user() and friends
sh: fix copy_from_user()
hexagon: fix strncpy_from_user() error return
mips: copy_from_user() must zero the destination on access_ok() failure
asm-generic: make copy_from_user() zero the destination properly
alpha: fix copy_from_user()
metag: copy_from_user() should zero the destination on access_ok() failure
parisc: fix copy_from_user()
openrisc: fix copy_from_user()
openrisc: fix the fix of copy_from_user()
mn10300: copy_from_user() should zero on access_ok() failure...
sparc32: fix copy_from_user()
ppc32: fix copy_from_user()
ia64: copy_from_user() should zero the destination on access_ok() failure
fix fault_in_multipages_...() on architectures with no-op access_ok()
fix memory leaks in tracing_buffers_splice_read()
arc: don't leak bits of kernel stack into coredump
Fix potential infoleak in older kernels
swapfile: fix memory corruption via malformed swapfile
coredump: fix unfreezable coredumping task
usb: dwc3: gadget: increment request->actual once
USB: validate wMaxPacketValue entries in endpoint descriptors
USB: fix typo in wMaxPacketSize validation
usb: xhci: Fix panic if disconnect
USB: serial: fix memleak in driver-registration error path
USB: kobil_sct: fix non-atomic allocation in write path
USB: serial: mos7720: fix non-atomic allocation in write path
USB: serial: mos7840: fix non-atomic allocation in write path
usb: renesas_usbhs: fix clearing the {BRDY,BEMP}STS condition
USB: change bInterval default to 10 ms
usb: gadget: fsl_qe_udc: signedness bug in qe_get_frame()
USB: serial: cp210x: fix hardware flow-control disable
usb: misc: legousbtower: Fix NULL pointer deference
usb: gadget: function: u_ether: don't starve tx request queue
USB: serial: cp210x: fix tiocmget error handling
usb: gadget: u_ether: remove interrupt throttling
usb: chipidea: move the lock initialization to core file
Fix USB CB/CBI storage devices with CONFIG_VMAP_STACK=y
ALSA: rawmidi: Fix possible deadlock with virmidi registration
ALSA: timer: fix NULL pointer dereference in read()/ioctl() race
ALSA: timer: fix division by zero after SNDRV_TIMER_IOCTL_CONTINUE
ALSA: timer: fix NULL pointer dereference on memory allocation failure
ALSA: ali5451: Fix out-of-bound position reporting
ALSA: pcm : Call kill_fasync() in stream lock
zfcp: fix fc_host port_type with NPIV
zfcp: fix ELS/GS request&response length for hardware data router
zfcp: close window with unblocked rport during rport gone
zfcp: retain trace level for SCSI and HBA FSF response records
zfcp: restore: Dont use 0 to indicate invalid LUN in rec trace
zfcp: trace on request for open and close of WKA port
zfcp: restore tracing of handle for port and LUN with HBA records
zfcp: fix D_ID field with actual value on tracing SAN responses
zfcp: fix payload trace length for SAN request&response
zfcp: trace full payload of all SAN records (req,resp,iels)
scsi: zfcp: spin_lock_irqsave() is not nestable
scsi: mpt3sas: Fix secure erase premature termination
scsi: mpt3sas: Unblock device after controller reset
scsi: mpt3sas: fix hang on ata passthrough commands
mpt2sas: Fix secure erase premature termination
scsi: megaraid_sas: Fix data integrity failure for JBOD (passthrough) devices
scsi: megaraid_sas: fix macro MEGASAS_IS_LOGICAL to avoid regression
scsi: ibmvfc: Fix I/O hang when port is not mapped
scsi: Fix use-after-free
scsi: arcmsr: Buffer overflow in arcmsr_iop_message_xfer()
scsi: scsi_debug: Fix memory leak if LBP enabled and module is unloaded
scsi: arcmsr: Send SYNCHRONIZE_CACHE command to firmware
ext4: validate that metadata blocks do not overlap superblock
ext4: avoid modifying checksum fields directly during checksum verification
ext4: use __GFP_NOFAIL in ext4_free_blocks()
ext4: reinforce check of i_dtime when clearing high fields of uid and gid
ext4: allow DAX writeback for hole punch
ext4: sanity check the block and cluster size at mount time
reiserfs: fix "new_insert_key may be used uninitialized ..."
reiserfs: Unlock superblock before calling reiserfs_quota_on_mount()
xfs: fix superblock inprogress check
libxfs: clean up _calc_dquots_per_chunk
btrfs: ensure that file descriptor used with subvol ioctls is a dir
ocfs2/dlm: fix race between convert and migration
ocfs2: fix start offset to ocfs2_zero_range_for_truncate()
ubifs: Fix assertion in layout_in_gaps()
ubifs: Fix xattr_names length in exit paths
UBIFS: Fix possible memory leak in ubifs_readdir()
ubifs: Abort readdir upon error
ubifs: Fix regression in ubifs_readdir()
UBI: fastmap: scrub PEB when bitflips are detected in a free PEB EC header
NFSv4.x: Fix a refcount leak in nfs_callback_up_net
NFSD: Using free_conn free connection
NFS: Don't drop CB requests with invalid principals
NFSv4: Open state recovery must account for file permission changes
fs/seq_file: fix out-of-bounds read
fs/super.c: fix race between freeze_super() and thaw_super()
isofs: Do not return EACCES for unknown filesystems
hostfs: Freeing an ERR_PTR in hostfs_fill_sb_common()
driver core: Delete an unnecessary check before the function call "put_device"
driver core: fix race between creating/querying glue dir and its cleanup
drm/radeon: fix radeon_move_blit on 32bit systems
drm: Reject page_flip for !DRIVER_MODESET
drm/radeon: Ensure vblank interrupt is enabled on DPMS transition to on
qxl: check for kmap failures
Input: i8042 - break load dependency between atkbd/psmouse and i8042
Input: i8042 - set up shared ps2_cmd_mutex for AUX ports
Input: ili210x - fix permissions on "calibrate" attribute
hwrng: exynos - Disable runtime PM on probe failure
hwrng: omap - Fix assumption that runtime_get_sync will always succeed
hwrng: omap - Only fail if pm_runtime_get_sync returns < 0
i2c-eg20t: fix race between i2c init and interrupt enable
em28xx-i2c: rt_mutex_trylock() returns zero on failure
i2c: core: fix NULL pointer dereference under race condition
i2c: at91: fix write transfers by clearing pending interrupt first
iio: accel: kxsd9: Fix raw read return
iio: accel: kxsd9: Fix scaling bug
thermal: hwmon: Properly report critical temperature in sysfs
cdc-acm: fix wrong pipe type on rx interrupt xfers
timers: Use proper base migration in add_timer_on()
EDAC: Increment correct counter in edac_inc_ue_error()
IB/ipoib: Fix memory corruption in ipoib cm mode connect flow
IB/core: Fix use after free in send_leave function
IB/ipoib: Don't allow MC joins during light MC flush
IB/mlx4: Fix incorrect MC join state bit-masking on SR-IOV
IB/mlx4: Fix create CQ error flow
IB/uverbs: Fix leak of XRC target QPs
IB/cm: Mark stale CM id's whenever the mad agent was unregistered
mtd: blkdevs: fix potential deadlock + lockdep warnings
mtd: pmcmsp-flash: Allocating too much in init_msp_flash()
mtd: nand: davinci: Reinitialize the HW ECC engine in 4bit hwctl
perf symbols: Fixup symbol sizes before picking best ones
perf: Tighten (and fix) the grouping condition
tty: Prevent ldisc drivers from re-using stale tty fields
tty: limit terminal size to 4M chars
tty: vt, fix bogus division in csi_J
vt: clear selection before resizing
drivers/vfio: Rework offsetofend()
include/stddef.h: Move offsetofend() from vfio.h to a generic kernel header
stddef.h: move offsetofend inside #ifndef/#endif guard, neaten
ipv6: don't call fib6_run_gc() until routing is ready
ipv6: split duplicate address detection and router solicitation timer
ipv6: move DAD and addrconf_verify processing to workqueue
ipv6: addrconf: fix dev refcont leak when DAD failed
ipv6: fix rtnl locking in setsockopt for anycast and multicast
ip6_gre: fix flowi6_proto value in ip6gre_xmit_other()
ipv6: correctly add local routes when lo goes up
ipv6: dccp: fix out of bound access in dccp_v6_err()
ipv6: dccp: add missing bind_conflict to dccp_ipv6_mapped
ip6_tunnel: Clear IP6CB in ip6tunnel_xmit()
ip6_tunnel: disable caching when the traffic class is inherited
net/irda: handle iriap_register_lsap() allocation failure
tcp: fix use after free in tcp_xmit_retransmit_queue()
tcp: properly scale window in tcp_v[46]_reqsk_send_ack()
tcp: fix overflow in __tcp_retransmit_skb()
tcp: fix wrong checksum calculation on MTU probing
tcp: take care of truncations done by sk_filter()
bonding: Fix bonding crash
net: ratelimit warnings about dst entry refcount underflow or overflow
mISDN: Support DR6 indication in mISDNipac driver
mISDN: Fixing missing validation in base_sock_bind()
net: disable fragment reassembly if high_thresh is set to zero
ipvs: count pre-established TCP states as active
iwlwifi: pcie: fix access to scratch buffer
svc: Avoid garbage replies when pc_func() returns rpc_drop_reply
brcmsmac: Free packet if dma_mapping_error() fails in dma_rxfill
brcmsmac: Initialize power in brcms_c_stf_ss_algo_channel_get()
brcmfmac: avoid potential stack overflow in brcmf_cfg80211_start_ap()
pstore: Fix buffer overflow while write offset equal to buffer size
net/mlx4_core: Allow resetting VF admin mac to zero
firewire: net: guard against rx buffer overflows
firewire: net: fix fragmented datagram_size off-by-one
netfilter: fix namespace handling in nf_log_proc_dostring
can: bcm: fix warning in bcm_connect/proc_register
net: fix sk_mem_reclaim_partial()
net: avoid sk_forward_alloc overflows
ipmr, ip6mr: fix scheduling while atomic and a deadlock with ipmr_get_route
packet: call fanout_release, while UNREGISTERING a netdev
net: sctp, forbid negative length
sctp: validate chunk len before actually using it
net: clear sk_err_soft in sk_clone_lock()
net: mangle zero checksum in skb_checksum_help()
dccp: do not send reset to already closed sockets
dccp: fix out of bound access in dccp_v4_err()
sctp: assign assoc_id earlier in __sctp_connect
neigh: check error pointer instead of NULL for ipv4_neigh_lookup()
ipv4: use new_gw for redirect neigh lookup
mac80211: fix purging multicast PS buffer queue
mac80211: discard multicast and 4-addr A-MSDUs
cfg80211: limit scan results cache size
mwifiex: printk() overflow with 32-byte SSIDs
ipv4: Set skb->protocol properly for local output
net: sky2: Fix shutdown crash
kaweth: fix firmware download
tracing: Move mutex to protect against resetting of seq data
kernel/fork: fix CLONE_CHILD_CLEARTID regression in nscd
Revert "ipc/sem.c: optimize sem_lock()"
cfq: fix starvation of asynchronous writes
drbd: Fix kernel_sendmsg() usage - potential NULL deref
lib/genalloc.c: start search from start of chunk
tools/vm/slabinfo: fix an unintentional printf
rcu: Fix soft lockup for rcu_nocb_kthread
ratelimit: fix bug in time interval by resetting right begin time
mfd: core: Fix device reference leak in mfd_clone_cell
PM / sleep: fix device reference leak in test_suspend
mmc: mxs: Initialize the spinlock prior to using it
mmc: block: don't use CMD23 with very old MMC cards
pstore/core: drop cmpxchg based updates
pstore/ram: Use memcpy_toio instead of memcpy
pstore/ram: Use memcpy_fromio() to save old buffer
mb86a20s: fix the locking logic
mb86a20s: fix demod settings
cx231xx: don't return error on success
cx231xx: fix GPIOs for Pixelview SBTVD hybrid
gpio: mpc8xxx: Correct irq handler function
uio: fix dmem_region_start computation
KEYS: Fix short sprintf buffer in /proc/keys show function
hv: do not lose pending heartbeat vmbus packets
staging: iio: ad5933: avoid uninitialized variable in error case
mei: bus: fix received data size check in NFC fixup
ACPI / APEI: Fix incorrect return value of ghes_proc()
PCI: Handle read-only BARs on AMD CS553x devices
tile: avoid using clocksource_cyc2ns with absolute cycle count
dm flakey: fix reads to be issued if drop_writes configured
mm,ksm: fix endless looping in allocating memory when ksm enable
can: dev: fix deadlock reported after bus-off
hwmon: (adt7411) set bit 3 in CFG1 register
mpi: Fix NULL ptr dereference in mpi_powm() [ver #3]
mfd: 88pm80x: Double shifting bug in suspend/resume
ASoC: omap-mcpdm: Fix irq resource handling
regulator: tps65910: Work around silicon erratum SWCZ010
dm: mark request_queue dead before destroying the DM device
fbdev/efifb: Fix 16 color palette entry calculation
metag: Only define atomic_dec_if_positive conditionally
Linux 3.10.105
Signed-off-by: Nathan Chancellor <natechancellor@gmail.com>
Conflicts:
arch/arm/mach-sa1100/generic.c
arch/arm64/kernel/traps.c
crypto/blkcipher.c
drivers/devfreq/devfreq.c
drivers/usb/dwc3/gadget.c
drivers/usb/gadget/u_ether.c
fs/ubifs/dir.c
include/net/if_inet6.h
lib/genalloc.c
net/ipv6/addrconf.c
net/ipv6/tcp_ipv6.c
net/wireless/scan.c
sound/core/timer.c
This reverts commit c0f4dfd4f9.
The above change allows the CPU to enter idle with RCU callbacks pending
for the durations specified in rcu_idle_gp_delay or rcu_idle_lazy_gp_delay.
This was causing a throughput decrease in synchronize_rcu(), as mentioned
by the original author.
These decreases were determined to be too large, as sub- 1 jiffy
synchronize_rcu() latency in cpu idle scenarios was desired.
Reverting this change change in order to start the force_quiescent_state()
state machine upon entering idle, rather than on the scheduler tick()
CRs-Fixed: 887035
Change-Id: I61865131dad751ec0fa14653b2b08b8bada4d061
Signed-off-by: Patrick Daly <pdaly@codeaurora.org>
When rcu_init() is called we already have slab working, allocating
bootmem at that point results in warnings and an allocation from
slab. This commit therefore changes alloc_bootmem_cpumask_var() to
alloc_cpumask_var() in rcu_bootup_announce_oddness(), which is called
from rcu_init().
Signed-off-by: Sasha Levin <sasha.levin@oracle.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
Tested-by: Robin Holt <holt@sgi.com>
[paulmck: convert to zalloc_cpumask_var(), as suggested by Yinghai Lu.]
Commit c0f4dfd4f (rcu: Make RCU_FAST_NO_HZ take advantage of numbered
callbacks) introduced a bug that can result in excessively long grace
periods. This bug reverse the senes of the "if" statement checking
for lazy callbacks, so that RCU takes a lazy approach when there are
in fact non-lazy callbacks. This can result in excessive boot, suspend,
and resume times.
This commit therefore fixes the sense of this "if" statement.
Reported-by: Borislav Petkov <bp@alien8.de>
Reported-by: Bjørn Mork <bjorn@mork.no>
Reported-by: Joerg Roedel <joro@8bytes.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Tested-by: Bjørn Mork <bjorn@mork.no>
Tested-by: Joerg Roedel <joro@8bytes.org>
Pull 'full dynticks' support from Ingo Molnar:
"This tree from Frederic Weisbecker adds a new, (exciting! :-) core
kernel feature to the timer and scheduler subsystems: 'full dynticks',
or CONFIG_NO_HZ_FULL=y.
This feature extends the nohz variable-size timer tick feature from
idle to busy CPUs (running at most one task) as well, potentially
reducing the number of timer interrupts significantly.
This feature got motivated by real-time folks and the -rt tree, but
the general utility and motivation of full-dynticks runs wider than
that:
- HPC workloads get faster: CPUs running a single task should be able
to utilize a maximum amount of CPU power. A periodic timer tick at
HZ=1000 can cause a constant overhead of up to 1.0%. This feature
removes that overhead - and speeds up the system by 0.5%-1.0% on
typical distro configs even on modern systems.
- Real-time workload latency reduction: CPUs running critical tasks
should experience as little jitter as possible. The last remaining
source of kernel-related jitter was the periodic timer tick.
- A single task executing on a CPU is a pretty common situation,
especially with an increasing number of cores/CPUs, so this feature
helps desktop and mobile workloads as well.
The cost of the feature is mainly related to increased timer
reprogramming overhead when a CPU switches its tick period, and thus
slightly longer to-idle and from-idle latency.
Configuration-wise a third mode of operation is added to the existing
two NOHZ kconfig modes:
- CONFIG_HZ_PERIODIC: [formerly !CONFIG_NO_HZ], now explicitly named
as a config option. This is the traditional Linux periodic tick
design: there's a HZ tick going on all the time, regardless of
whether a CPU is idle or not.
- CONFIG_NO_HZ_IDLE: [formerly CONFIG_NO_HZ=y], this turns off the
periodic tick when a CPU enters idle mode.
- CONFIG_NO_HZ_FULL: this new mode, in addition to turning off the
tick when a CPU is idle, also slows the tick down to 1 Hz (one
timer interrupt per second) when only a single task is running on a
CPU.
The .config behavior is compatible: existing !CONFIG_NO_HZ and
CONFIG_NO_HZ=y settings get translated to the new values, without the
user having to configure anything. CONFIG_NO_HZ_FULL is turned off by
default.
This feature is based on a lot of infrastructure work that has been
steadily going upstream in the last 2-3 cycles: related RCU support
and non-periodic cputime support in particular is upstream already.
This tree adds the final pieces and activates the feature. The pull
request is marked RFC because:
- it's marked 64-bit only at the moment - the 32-bit support patch is
small but did not get ready in time.
- it has a number of fresh commits that came in after the merge
window. The overwhelming majority of commits are from before the
merge window, but still some aspects of the tree are fresh and so I
marked it RFC.
- it's a pretty wide-reaching feature with lots of effects - and
while the components have been in testing for some time, the full
combination is still not very widely used. That it's default-off
should reduce its regression abilities and obviously there are no
known regressions with CONFIG_NO_HZ_FULL=y enabled either.
- the feature is not completely idempotent: there is no 100%
equivalent replacement for a periodic scheduler/timer tick. In
particular there's ongoing work to map out and reduce its effects
on scheduler load-balancing and statistics. This should not impact
correctness though, there are no known regressions related to this
feature at this point.
- it's a pretty ambitious feature that with time will likely be
enabled by most Linux distros, and we'd like you to make input on
its design/implementation, if you dislike some aspect we missed.
Without flaming us to crisp! :-)
Future plans:
- there's ongoing work to reduce 1Hz to 0Hz, to essentially shut off
the periodic tick altogether when there's a single busy task on a
CPU. We'd first like 1 Hz to be exposed more widely before we go
for the 0 Hz target though.
- once we reach 0 Hz we can remove the periodic tick assumption from
nr_running>=2 as well, by essentially interrupting busy tasks only
as frequently as the sched_latency constraints require us to do -
once every 4-40 msecs, depending on nr_running.
I am personally leaning towards biting the bullet and doing this in
v3.10, like the -rt tree this effort has been going on for too long -
but the final word is up to you as usual.
More technical details can be found in Documentation/timers/NO_HZ.txt"
* 'timers-nohz-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: (39 commits)
sched: Keep at least 1 tick per second for active dynticks tasks
rcu: Fix full dynticks' dependency on wide RCU nocb mode
nohz: Protect smp_processor_id() in tick_nohz_task_switch()
nohz_full: Add documentation.
cputime_nsecs: use math64.h for nsec resolution conversion helpers
nohz: Select VIRT_CPU_ACCOUNTING_GEN from full dynticks config
nohz: Reduce overhead under high-freq idling patterns
nohz: Remove full dynticks' superfluous dependency on RCU tree
nohz: Fix unavailable tick_stop tracepoint in dynticks idle
nohz: Add basic tracing
nohz: Select wide RCU nocb for full dynticks
nohz: Disable the tick when irq resume in full dynticks CPU
nohz: Re-evaluate the tick for the new task after a context switch
nohz: Prepare to stop the tick on irq exit
nohz: Implement full dynticks kick
nohz: Re-evaluate the tick from the scheduler IPI
sched: New helper to prevent from stopping the tick in full dynticks
sched: Kick full dynticks CPU that have more than one task enqueued.
perf: New helper to prevent full dynticks CPUs from stopping tick
perf: Kick full dynticks CPU if events rotation is needed
...
The full dynticks tree needs the latest RCU and sched
upstream updates in order to fix some dependencies.
Merge a common upstream merge point that has these
updates.
Conflicts:
include/linux/perf_event.h
kernel/rcutree.h
kernel/rcutree_plugin.h
Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
We need full dynticks CPU to also be RCU nocb so
that we don't have to keep the tick to handle RCU
callbacks.
Make sure the range passed to nohz_full= boot
parameter is a subset of rcu_nocbs=
The CPUs that fail to meet this requirement will be
excluded from the nohz_full range. This is checked
early in boot time, before any CPU has the opportunity
to stop its tick.
Suggested-by: Steven Rostedt <rostedt@goodmis.org>
Reviewed-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Chris Metcalf <cmetcalf@tilera.com>
Cc: Christoph Lameter <cl@linux.com>
Cc: Geoff Levand <geoff@infradead.org>
Cc: Gilad Ben Yossef <gilad@benyossef.com>
Cc: Hakan Akkan <hakanakkan@gmail.com>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Kevin Hilman <khilman@linaro.org>
Cc: Li Zhong <zhong@linux.vnet.ibm.com>
Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Paul Gortmaker <paul.gortmaker@windriver.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Adaptive-ticks CPUs inform RCU when they enter kernel mode, but they do
not necessarily turn the scheduler-clock tick back on. This state of
affairs could result in RCU waiting on an adaptive-ticks CPU running
for an extended period in kernel mode. Such a CPU will never run the
RCU state machine, and could therefore indefinitely extend the RCU state
machine, sooner or later resulting in an OOM condition.
This patch, inspired by an earlier patch by Frederic Weisbecker, therefore
causes RCU's force-quiescent-state processing to check for this condition
and to send an IPI to CPUs that remain in that state for too long.
"Too long" currently means about three jiffies by default, which is
quite some time for a CPU to remain in the kernel without blocking.
The rcu_tree.jiffies_till_first_fqs and rcutree.jiffies_till_next_fqs
sysfs variables may be used to tune "too long" if needed.
Reported-by: Frederic Weisbecker <fweisbec@gmail.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Chris Metcalf <cmetcalf@tilera.com>
Cc: Christoph Lameter <cl@linux.com>
Cc: Geoff Levand <geoff@infradead.org>
Cc: Gilad Ben Yossef <gilad@benyossef.com>
Cc: Hakan Akkan <hakanakkan@gmail.com>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Kevin Hilman <khilman@linaro.org>
Cc: Li Zhong <zhong@linux.vnet.ibm.com>
Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Paul Gortmaker <paul.gortmaker@windriver.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
doc.2013.03.12a: Documentation changes.
fixes.2013.03.13a: Miscellaneous fixes.
idlenocb.2013.03.26b: Remove restrictions on no-CBs CPUs, make
RCU_FAST_NO_HZ take advantage of numbered callbacks, add
callback acceleration based on numbered callbacks.
CPUs going idle will need to record the need for a future grace
period, but won't actually need to block waiting on it. This commit
therefore splits rcu_start_future_gp(), which does the recording, from
rcu_nocb_wait_gp(), which now invokes rcu_start_future_gp() to do the
recording, after which rcu_nocb_wait_gp() does the waiting.
Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
CPUs going idle need to be able to indicate their need for future grace
periods. A mechanism for doing this already exists for no-callbacks
CPUs, so the idea is to re-use that mechanism. This commit therefore
moves the ->n_nocb_gp_requests field of the rcu_node structure out from
under the CONFIG_RCU_NOCB_CPU #ifdef and renames it to ->need_future_gp.
Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
If CPUs are to give prior notice of needed grace periods, it will be
necessary to invoke rcu_start_gp() without dropping the root rcu_node
structure's ->lock. This commit takes a second step in this direction
by moving the release of this lock to rcu_start_gp()'s callers.
Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Dyntick-idle CPUs need to be able to pre-announce their need for grace
periods. This can be done using something similar to the mechanism used
by no-CB CPUs to announce their need for grace periods. This commit
moves in this direction by renaming the no-CBs grace-period event tracing
to suit the new future-grace-period needs.
Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Because RCU callbacks are now associated with the number of the grace
period that they must wait for, CPUs can now take advance callbacks
corresponding to grace periods that ended while a given CPU was in
dyntick-idle mode. This eliminates the need to try forcing the RCU
state machine while entering idle, thus reducing the CPU intensiveness
of RCU_FAST_NO_HZ, which should increase its energy efficiency.
Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
RCU_FAST_NO_HZ operation is controlled by four compile-time C-preprocessor
macros, but some use cases benefit greatly from runtime adjustment,
particularly when tuning devices. This commit therefore creates the
corresponding sysfs entries.
Reported-by: Robin Randhawa <robin.randhawa@arm.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Currently, the per-no-CBs-CPU kthreads are named "rcuo" followed by
the CPU number, for example, "rcuo". This is problematic given that
there are either two or three RCU flavors, each of which gets a per-CPU
kthread with exactly the same name. This commit therefore introduces
a one-letter abbreviation for each RCU flavor, namely 'b' for RCU-bh,
'p' for RCU-preempt, and 's' for RCU-sched. This abbreviation is used
to distinguish the "rcuo" kthreads, for example, for CPU 0 we would have
"rcuob/0", "rcuop/0", and "rcuos/0".
Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Tested-by: Dietmar Eggemann <dietmar.eggemann@arm.com>
Currently, the no-CBs kthreads do repeated timed waits for grace periods
to elapse. This is crude and energy inefficient, so this commit allows
no-CBs kthreads to specify exactly which grace period they are waiting
for and also allows them to block for the entire duration until the
desired grace period completes.
Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Currently, the only way to specify no-CBs CPUs is via the rcu_nocbs
kernel command-line parameter. This is inconvenient in some cases,
particularly for randconfig testing, so this commit adds a new set of
kernel configuration parameters. CONFIG_RCU_NOCB_CPU_NONE (the default)
retains the old behavior, CONFIG_RCU_NOCB_CPU_ZERO offloads callback
processing from CPU 0 (along with any other CPUs specified by the
rcu_nocbs boot-time parameter), and CONFIG_RCU_NOCB_CPU_ALL offloads
callback processing from all CPUs.
Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
If RCU's softirq handler is prevented from executing, an RCU CPU stall
warning can result. Ways to prevent RCU's softirq handler from executing
include: (1) CPU spinning with interrupts disabled, (2) infinite loop
in some softirq handler, and (3) in -rt kernels, an infinite loop in a
set of real-time threads running at priorities higher than that of RCU's
softirq handler.
Because this situation can be difficult to track down, this commit causes
the count of RCU softirq handler invocations to be printed with RCU
CPU stall warnings. This information does require some interpretation,
as now documented in Documentation/RCU/stallwarn.txt.
Reported-by: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Tested-by: Paul Gortmaker <paul.gortmaker@windriver.com>
Currently, CPU 0 is constrained to not be a no-CBs CPU, and furthermore
at least one no-CBs CPU must remain online at any given time. These
restrictions are problematic in some situations, such as cases where
all CPUs must run a real-time workload that needs to be insulated from
OS jitter and latencies due to RCU callback invocation. This commit
therefore provides no-CBs CPUs a (very crude and energy-inefficient)
way to start and to wait for grace periods independently of the normal
RCU callback mechanisms. This approach allows any or all of the CPUs to
be designated as no-CBs CPUs, and allows any proper subset of the CPUs
(whether no-CBs CPUs or not) to be offlined.
This commit also provides a fix for a locking bug spotted by Xie
ChanglongX <changlongx.xie@intel.com>.
Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
The as-documented rcu_nocb_poll will fail to enable this feature
for two reasons. (1) there is an extra "s" in the documented
name which is not in the code, and (2) since it uses module_param,
it really is expecting a prefix, akin to "rcutree.fanout_leaf"
and the prefix isn't documented.
However, there are several reasons why we might not want to
simply fix the typo and add the prefix:
1) we'd end up with rcutree.rcu_nocb_poll, and rather probably make
a change to rcutree.nocb_poll
2) if we did #1, then the prefix wouldn't be consistent with the
rcu_nocbs=<cpumap> parameter (i.e. one with, one without prefix)
3) the use of module_param in a header file is less than desired,
since it isn't immediately obvious that it will get processed
via rcutree.c and get the prefix from that (although use of
module_param_named() could clarify that.)
4) the implied export of /sys/module/rcutree/parameters/rcu_nocb_poll
data to userspace via module_param() doesn't really buy us anything,
as it is read-only and we can tell if it is enabled already without
it, since there is a printk at early boot telling us so.
In light of all that, just change it from a module_param() to an
early_setup() call, and worry about adding it to /sys later on if
we decide to allow a dynamic setting of it.
Also change the variable to be tagged as read_mostly, since it
will only ever be fiddled with at most, once at boot.
Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
The wait_event() at the head of the rcu_nocb_kthread() can result in
soft-lockup complaints if the CPU in question does not register RCU
callbacks for an extended period. This commit therefore changes
the wait_event() to a wait_event_interruptible().
Reported-by: Frederic Weisbecker <fweisbec@gmail.com>
Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Currently, callback invocations from callback-free CPUs are accounted to
the CPU that registered the callback, but using the same field that is
used for normal callbacks. This makes it impossible to determine from
debugfs output whether callbacks are in fact being diverted. This commit
therefore adds a separate ->n_nocbs_invoked field in the rcu_data structure
in which diverted callback invocations are counted. RCU's debugfs tracing
still displays normal callback invocations using ci=, but displayed
diverted callbacks with nci=.
Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
RCU callback execution can add significant OS jitter and also can
degrade both scheduling latency and, in asymmetric multiprocessors,
energy efficiency. This commit therefore adds the ability for selected
CPUs ("rcu_nocbs=" boot parameter) to have their callbacks offloaded
to kthreads. If the "rcu_nocb_poll" boot parameter is also specified,
these kthreads will do polling, removing the need for the offloaded
CPUs to do wakeups. At least one CPU must be doing normal callback
processing: currently CPU 0 cannot be selected as a no-CBs CPU.
In addition, attempts to offline the last normal-CBs CPU will fail.
This feature was inspired by Jim Houston's and Joe Korty's JRCU, and
this commit includes fixes to problems located by Fengguang Wu's
kbuild test robot.
[ paulmck: Added gfp.h include file as suggested by Fengguang Wu. ]
Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
urgent.2012.10.27a: Fix for RCU user-mode transition (already in -tip).
doc.2012.11.08a: Documentation updates, most notably codifying the
memory-barrier guarantees inherent to grace periods.
fixes.2012.11.13a: Miscellaneous fixes.
srcu.2012.10.27a: Allow statically allocated and initialized srcu_struct
structures (courtesy of Lai Jiangshan).
stall.2012.11.13a: Add more diagnostic information to RCU CPU stall
warnings, also decrease from 60 seconds to 21 seconds.
hotplug.2012.11.08a: Minor updates to CPU hotplug handling.
tracing.2012.11.08a: Improved debugfs tracing, courtesy of Michael Wang.
idle.2012.10.24a: Updates to RCU idle/adaptive-idle handling, including
a boot parameter that maps normal grace periods to expedited.
Resolved conflict in kernel/rcutree.c due to side-by-side change.
This commit explicitly states the memory-ordering properties of the
RCU grace-period primitives. Although these properties were in some
sense implied by the fundmental property of RCU ("a grace period must
wait for all pre-existing RCU read-side critical sections to complete"),
stating it explicitly will be a great labor-saving device.
Reported-by: Oleg Nesterov <oleg@redhat.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Oleg Nesterov <oleg@redhat.com>
The ->onofflock field in the rcu_state structure at one time synchronized
CPU-hotplug operations for RCU. However, its scope has decreased over time
so that it now only protects the lists of orphaned RCU callbacks. This
commit therefore renames it to ->orphan_lock to reflect its current use.
Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
There have been some embedded applications that would benefit from
use of expedited grace-period primitives. In some ways, this is
similar to synchronize_net() doing either a normal or an expedited
grace period depending on lock state, but with control outside of
the kernel.
This commit therefore adds rcu_expedited boot and sysfs parameters
that cause the kernel to substitute expedited primitives for the
normal grace-period primitives.
[ paulmck: Add trace/event/rcu.h to kernel/srcu.c to avoid build error.
Get rid of infinite loop through contention path.]
Signed-off-by: Antti P Miettinen <amiettinen@nvidia.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
The current implementation of RCU_FAST_NO_HZ tries reasonably hard to rid
the current CPU of RCU callbacks. This is appropriate when the CPU is
entering idle, where it doesn't have much useful to do anyway, but is most
definitely not what you want when transitioning to user-mode execution.
This commit therefore detects the adaptive-tick case, and refrains from
burning CPU time getting rid of RCU callbacks in that case.
Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
The conflicts between kernel/rcutree.h and kernel/rcutree_plugin.h
were due to adjacent insertions and deletions, which were resolved
by simply accepting the changes on both branches.
bigrt.2012.09.23a contains additional commits to reduce scheduling latency
from RCU on huge systems (many hundrends or thousands of CPUs).
doctorture.2012.09.23a contains documentation changes and rcutorture fixes.
fixes.2012.09.23a contains miscellaneous fixes.
hotplug.2012.09.23a contains CPU-hotplug-related changes.
idle.2012.09.23a fixes architectures for which RCU no longer considered
the idle loop to be a quiescent state due to earlier
adaptive-dynticks changes. Affected architectures are alpha,
cris, frv, h8300, m32r, m68k, mn10300, parisc, score, xtensa,
and ia64.
The print_cpu_stall_fast_no_hz() function attempts to print -1 when
the ->idle_gp_timer is not pending, but unsigned arithmetic causes it
to instead print ULONG_MAX, which is 4294967295 on 32-bit systems and
18446744073709551615 on 64-bit systems. Neither of these are the most
reader-friendly values, so this commit instead causes "timer not pending"
to be printed when ->idle_gp_timer is not pending.
Reported-by: Paul Walmsley <paul@pwsan.com>
Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
The rcu_print_detail_task_stall_rnp() function invokes
rcu_preempt_blocked_readers_cgp() to verify that there are some preempted
RCU readers blocking the current grace period outside of the protection
of the rcu_node structure's ->lock. This means that the last blocked
reader might exit its RCU read-side critical section and remove itself
from the ->blkd_tasks list before the ->lock is acquired, resulting in
a segmentation fault when the subsequent code attempts to dereference
the now-NULL gp_tasks pointer.
This commit therefore moves the test under the lock. This will not
have measurable effect on lock contention because this code is invoked
only when printing RCU CPU stall warnings, in other words, in the common
case, never.
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
The increment_cpu_stall_ticks() function listed each RCU flavor
explicitly, with an ifdef to handle preemptible RCU. This commit
therefore applies for_each_rcu_flavor() to save a line of code.
Because this commit switches from a code-based enumeration of the
flavors of RCU to an rcu_state-list-based enumeration, it is no longer
possible to apply __get_cpu_var() to the per-CPU rcu_data structures.
We instead use __this_cpu_var() on the rcu_state structure's ->rda field
that references the corresponding rcu_data structures.
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Commit 1217ed1b (rcu: permit rcu_read_unlock() to be called while holding
runqueue locks) made rcu_initiate_boost() restore irq state when releasing
the rcu_node structure's ->lock, but failed to update the header comment
accordingly. This commit therefore brings the header comment up to date.
Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
The rcu_preempt_offline_tasks() moves all tasks queued on a given leaf
rcu_node structure to the root rcu_node, which is done when the last CPU
corresponding the the leaf rcu_node structure goes offline. Now that
RCU-preempt's synchronize_rcu_expedited() implementation blocks CPU-hotplug
operations during the initialization of each rcu_node structure's
->boost_tasks pointer, rcu_preempt_offline_tasks() can do a better job
of setting the root rcu_node's ->boost_tasks pointer.
The key point is that rcu_preempt_offline_tasks() runs as part of the
CPU-hotplug process, so that a concurrent synchronize_rcu_expedited()
is guaranteed to either have not started on the one hand (in which case
there is no boosting on behalf of the expedited grace period) or to be
completely initialized on the other (in which case, in the absence of
other priority boosting, all ->boost_tasks pointers will be initialized).
Therefore, if rcu_preempt_offline_tasks() finds that the ->boost_tasks
pointer is equal to the ->exp_tasks pointer, it can be sure that it is
correctly placed.
In the case where there was boosting ongoing at the time that the
synchronize_rcu_expedited() function started, different nodes might start
boosting the tasks blocking the expedited grace period at different times.
In this mixed case, the root node will either be boosting tasks for
the expedited grace period already, or it will start as soon as it gets
done boosting for the normal grace period -- but in this latter case,
the root node's tasks needed to be boosted in any case.
This commit therefore adds a check of the ->boost_tasks pointer against
the ->exp_tasks pointer to the list that prevents updating ->boost_tasks.
Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
When rcu_preempt_offline_tasks() clears tasks from a leaf rcu_node
structure, it does not NULL out the structure's ->boost_tasks field.
This commit therefore fixes this issue.
Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
The current quiescent-state detection algorithm is needlessly
complex. It records the grace-period number corresponding to
the quiescent state at the time of the quiescent state, which
works, but it seems better to simply erase any record of previous
quiescent states at the time that the CPU notices the new grace
period. This has the further advantage of removing another piece
of RCU for which lockless reasoning is required.
Therefore, this commit makes this change.
Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
The synchronize_rcu_expedited() function disables interrupts across a
scan of all leaf rcu_node structures, which is not good for real-time
scheduling latency on large systems (hundreds or especially thousands
of CPUs). This commit therefore holds off CPU-hotplug operations using
get_online_cpus(), and removes the prior acquisiion of the ->onofflock
(which required disabling interrupts).
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
In the C language, signed overflow is undefined. It is true that
twos-complement arithmetic normally comes to the rescue, but if the
compiler can subvert this any time it has any information about the values
being compared. For example, given "if (a - b > 0)", if the compiler
has enough information to realize that (for example) the value of "a"
is positive and that of "b" is negative, the compiler is within its
rights to optimize to a simple "if (1)", which might not be what you want.
This commit therefore converts synchronize_rcu_expedited()'s work-done
detection counter from signed to unsigned.
Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
As the first step towards allowing quiescent-state forcing to be
preemptible, this commit moves RCU quiescent-state forcing into the
same kthread that is now used to initialize and clean up after grace
periods. This is yet another step towards keeping scheduling
latency down to a dull roar.
Updated to change from raw_spin_lock_irqsave() to raw_spin_lock_irq()
and to remove the now-unused rcu_state structure fields as suggested by
Peter Zijlstra.
Reported-by: Mike Galbraith <mgalbraith@suse.de>
Reported-by: Dimitri Sivanich <sivanich@sgi.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
In kernels built with CONFIG_RCU_FAST_NO_HZ=y, CPUs can accumulate a
large number of lazy callbacks, which as the name implies will be slow
to be invoked. This can be a problem on small-memory systems, where the
default 6-second sleep for CPUs having only lazy RCU callbacks could well
be fatal. This commit therefore installs an OOM hander that ensures that
every CPU with lazy callbacks has at least one non-lazy callback, in turn
ensuring timely advancement for these callbacks.
Updated to fix bug that disabled OOM killing, noted by Lai Jiangshan.
Updated to push the for_each_rcu_flavor() loop into rcu_oom_notify_cpu(),
thus reducing the number of IPIs, as suggested by Steven Rostedt. Also
to make the for_each_online_cpu() loop be preemptible. (Later, it might
be good to use smp_call_function(), as suggested by Peter Zijlstra.)
Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Tested-by: Sasha Levin <levinsasha928@gmail.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
Bring RCU into the new-age CPU-hotplug fold by modifying RCU's per-CPU
kthread code to use the new smp_hotplug_thread facility.
[ tglx: Adapted it to use callbacks and to the simplified rcu yield ]
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Srivatsa S. Bhat <srivatsa.bhat@linux.vnet.ibm.com>
Cc: Rusty Russell <rusty@rustcorp.com.au>
Cc: Namhyung Kim <namhyung@kernel.org>
Link: http://lkml.kernel.org/r/20120716103948.673354828@linutronix.de
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
The rcu_yield() code is amazing. It's there to avoid starvation of the
system when lots of (boosting) work is to be done.
Now looking at the code it's functionality is:
Make the thread SCHED_OTHER and very nice, i.e. get it out of the way
Arm a timer with 2 ticks
schedule()
Now if the system goes idle the rcu task returns, regains SCHED_FIFO
and plugs on. If the systems stays busy the timer fires and wakes a
per node kthread which in turn makes the per cpu thread SCHED_FIFO and
brings it back on the cpu. For the boosting thread the "make it FIFO"
bit is missing and it just runs some magic boost checks. Now this is a
lot of code with extra threads and complexity.
It's way simpler to let the tasks when they detect overload schedule
away for 2 ticks and defer the normal wakeup as long as they are in
yielded state and the cpu is not idle.
That solves the same problem and the only difference is that when the
cpu goes idle it's not guaranteed that the thread returns right away,
but it won't be longer out than two ticks, so no harm is done. If
that's an issue than it is way simpler just to wake the task from
idle as RCU has callbacks there anyway.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Srivatsa S. Bhat <srivatsa.bhat@linux.vnet.ibm.com>
Cc: Rusty Russell <rusty@rustcorp.com.au>
Cc: Namhyung Kim <namhyung@kernel.org>
Reviewed-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Link: http://lkml.kernel.org/r/20120716103948.131256723@linutronix.de
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
The Linux kernel coding style says that single-statement blocks should
omit curly braces unless the other leg of the "if" statement has
multiple statements, in which case the curly braces should be included.
This commit fixes RCU's violations of this rule.
Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
bigrtm: First steps towards getting RCU out of the way of
tens-of-microseconds real-time response on systems compiled
with NR_CPUS=4096. Also cleanups for and increased concurrency
of rcu_barrier() family of primitives.
doctorture: rcutorture and documentation improvements.
fixes: Miscellaneous fixes.
fnh: RCU_FAST_NO_HZ fixes and improvements.
If the nohz= boot parameter disables nohz, then RCU_FAST_NO_HZ needs to
also disable itself. This commit therefore checks for tick_nohz_enabled
being zero, disabling rcu_prepare_for_idle() if so. This commit assumes
that tick_nohz_enabled can change at runtime: If this is not the case,
then a simpler approach suffices.
Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Nathan Chancellor