- 28 Apr, 2014 40 commits
-
-
Gavin Shan authored
The issue was detected in a bit complicated test case where we have multiple hierarchical PEs shown as following figure: +-----------------+ | PE#3 p2p#0 | | p2p#1 | +-----------------+ | +-----------------+ | PE#4 pdev#0 | | pdev#1 | +-----------------+ PE#4 (have 2 PCI devices) is the child of PE#3, which has 2 p2p bridges. We accidentally had less-known scenario: PE#4 was removed permanently from the system because of permanent failure (e.g. exceeding the max allowd failure times in last hour), then we detects EEH errors on PE#3 and tried to recover it. However, eeh_dev instances for pdev#0/1 were not detached from PE#4, which was still connected to PE#3. All of that was because of the fact that we rely on count-based pcibios_release_device(), which isn't reliable enough. When doing recovery for PE#3, we still apply hotplug on PE#4 and pdev#0/1, which are not valid any more. Eventually, we run into kernel crash. The patch fixes above issue from two aspects. For unplug, we simply skip those permanently removed PE, whose state is (EEH_PE_STATE_ISOLATED && !EEH_PE_STATE_RECOVERING) and its frozen count should be greater than EEH_MAX_ALLOWED_FREEZES. For plug, we marked all permanently removed EEH devices with EEH_DEV_REMOVED and return 0xFF's on read its PCI config so that PCI core will omit them. Signed-off-by:Gavin Shan <gwshan@linux.vnet.ibm.com> Signed-off-by:
Benjamin Herrenschmidt <benh@kernel.crashing.org>
-
Gavin Shan authored
The patch introduces bootarg "eeh=off" to disable EEH functinality. Also, it creates /sys/kerenl/debug/powerpc/eeh_enable to disable or enable EEH functionality. By default, we have the functionality enabled. For PowerNV platform, we will restore to have the conventional mechanism of clearing frozen PE during PCI config access if we're going to disable EEH functionality. Conversely, we will rely on EEH for error recovery. The patch also fixes the issue that we missed to cover the case of disabled EEH functionality in function ioda_eeh_event(). Those events driven by interrupt should be cleared to avoid endless reporting. Signed-off-by:
-
Gavin Shan authored
There're 2 EEH subsystem variables: eeh_subsystem_enabled and eeh_probe_mode. We needn't maintain 2 variables and we can just have one variable and introduce different flags. The patch also introduces additional flag EEH_FORCE_DISABLE, which will be used to disable EEH subsystem via boot parameter ("eeh=off") in future. Besides, the patch also introduces flag EEH_ENABLED, which is changed to disable or enable EEH functionality on the fly through debugfs entry in future. With the patch applied, the creteria to check the enabled EEH functionality is changed to: !EEH_FORCE_DISABLED && EEH_ENABLED : Enabled Other cases : Disabled Signed-off-by: -
Gavin Shan authored
When calling into eeh_gather_pci_data() on pSeries platform, we possiblly don't have pci_dev instance yet, but eeh_dev is always ready. So we use cached capability from eeh_dev instead of pci_dev for log dump there. In order to keep things unified, we also cache PCI capability positions to eeh_dev for PowerNV as well. Signed-off-by:
-
Gavin Shan authored
The patch replaces printk(KERN_WARNING ...) with pr_warn() in the function eeh_gather_pci_data(). Signed-off-by:
-
Gavin Shan authored
We have suffered recrusive frozen PE a lot, which was caused by IO accesses during the PE reset. Ben came up with the good idea to keep frozen PE until recovery (BAR restore) gets done. With that, IO accesses during PE reset are dropped by hardware and wouldn't incur the recrusive frozen PE any more. The patch implements the idea. We don't clear the frozen state until PE reset is done completely. During the period, the EEH core expects unfrozen state from backend to keep going. So we have to reuse EEH_PE_RESET flag, which has been set during PE reset, to return normal state from backend. The side effect is we have to clear frozen state for towice (PE reset and clear it explicitly), but that's harmless. We have some limitations on pHyp. pHyp doesn't allow to enable IO or DMA for unfrozen PE. So we don't enable them on unfrozen PE in eeh_pci_enable(). We have to enable IO before grabbing logs on pHyp. Otherwise, 0xFF's is always returned from PCI config space. Also, we had wrong return value from eeh_pci_enable() for EEH_OPT_THAW_DMA case. The patch fixes it too. Signed-off-by:
-
Gavin Shan authored
For EEH PowerNV backends, they need use their own PCI config accesors as the normal one could be blocked during PE reset. The patch also removes necessary parameter "hose" for the function ioda_eeh_bridge_reset(). Signed-off-by:
-
Gavin Shan authored
We've observed multiple PE reset failures because of PCI-CFG access during that period. Potentially, some device drivers can't support EEH very well and they can't put the device to motionless state before PE reset. So those device drivers might produce PCI-CFG accesses during PE reset. Also, we could have PCI-CFG access from user space (e.g. "lspci"). Since access to frozen PE should return 0xFF's, we can block PCI-CFG access during the period of PE reset so that we won't get recrusive EEH errors. The patch adds flag EEH_PE_RESET, which is kept during PE reset. The PowerNV/pSeries PCI-CFG accessors reuse the flag to block PCI-CFG accordingly. Signed-off-by:
-
Gavin Shan authored
When doing PE reset, EEH_PE_ISOLATED is cleared unconditionally. However, We should remove that if the PE reset has cleared the frozen state successfully. Otherwise, the flag should be kept. The patch fixes the issue. Signed-off-by:
-
Gavin Shan authored
For some fields (e.g. LEM, MMIO, DMA) in PHB diag-data dump, it's meaningless to print them if they have non-zero value in the corresponding mask registers because we always have non-zero values in the mask registers. The patch only prints those fieds if we have non-zero values in the primary registers (e.g. LEM, MMIO, DMA status) so that we can save couple of lines. The patch also removes unnecessary spare line before "brdgCtl:" and two leading spaces as prefix in each line as Ben suggested. Signed-off-by:
-
Gavin Shan authored
The flag PNV_EEH_STATE_ENABLED is put into pnv_phb::eeh_state, which is protected by CONFIG_EEH. We needn't that. Instead, we can have pnv_phb::flags and maintain all flags there, which is the purpose of the patch. The patch also renames PNV_EEH_STATE_ENABLED to PNV_PHB_FLAG_EEH. Signed-off-by:
-
Gavin Shan authored
The PHB state PNV_EEH_STATE_REMOVED maintained in pnv_phb isn't so useful any more and it's duplicated to EEH_PE_ISOLATED. The patch replaces PNV_EEH_STATE_REMOVED with EEH_PE_ISOLATED. Signed-off-by:
-
Gavin Shan authored
The PE state (for eeh_pe instance) EEH_PE_PHB_DEAD is duplicate to EEH_PE_ISOLATED. Originally, those PHBs (PHB PE) with EEH_PE_PHB_DEAD would be removed from the system. However, it's safe to replace that with EEH_PE_ISOLATED. The patch also clear EEH_PE_RECOVERING after fenced PHB has been handled, either failure or success. It makes the PHB PE state consistent with: PHB functions normally NONE PHB has been removed EEH_PE_ISOLATED PHB fenced, recovery in progress EEH_PE_ISOLATED | RECOVERING Signed-off-by:
-
Alistair Popple authored
This patch fixes this section mismatch: WARNING: vmlinux.o(.text+0x1efc4): Section mismatch in reference from the function apm821xx_pciex_init_port_hw() to the function .init.text:ppc4xx_pciex_wait_on_sdr.isra.9() The function apm821xx_pciex_init_port_hw() references the function __init ppc4xx_pciex_wait_on_sdr.isra.9(). This is often because apm821xx_pciex_init_port_hw lacks a __init annotation or the annotation of ppc4xx_pciex_wait_on_sdr.isra.9 is wrong. apm821xx_pciex_init_port_hw is only referenced by a struct in __initdata, so it should be safe to add __init to apm821xx_pciex_init_port_hw. Signed-off-by:
Alistair Popple <alistair@popple.id.au> Signed-off-by:
-
Preeti U Murthy authored
When the guest cedes the vcpu or the vcpu has no guest to run it naps. Clear the runlatch bit of the vcpu before napping to indicate an idle cpu. Signed-off-by:
Preeti U Murthy <preeti@linux.vnet.ibm.com> Acked-by:
Paul Mackerras <paulus@samba.org> Reviewed-by:
Srivatsa S. Bhat <srivatsa.bhat@linux.vnet.ibm.com> Signed-off-by:
-
Preeti U Murthy authored
The secondary threads in the core are kept offline before launching guests in kvm on powerpc: "371fefd6:KVM: PPC: Allow book3s_hv guests to use SMT processor modes." Hence their runlatch bits are cleared. When the secondary threads are called in to start a guest, their runlatch bits need to be set to indicate that they are busy. The primary thread has its runlatch bit set though, but there is no harm in setting this bit once again. Hence set the runlatch bit for all threads before they start guest. Signed-off-by:
-
Preeti U Murthy authored
Up until now we have been setting the runlatch bits for a busy CPU and clearing it when a CPU enters idle state. The runlatch bit has thus been consistent with the utilization of a CPU as long as the CPU is online. However when a CPU is hotplugged out the runlatch bit is not cleared. It needs to be cleared to indicate an unused CPU. Hence this patch has the runlatch bit cleared for an offline CPU just before entering an idle state and sets it immediately after it exits the idle state. Signed-off-by:
-
Li Zhong authored
While testing memory hot-remove, I found following dead lock: Process #1141 is drmgr, trying to remove some memory, i.e. memory499. It holds the memory_hotplug_mutex, and blocks when trying to remove file "online" under dir memory499, in kernfs_drain(), at wait_event(root->deactivate_waitq, atomic_read(&kn->active) == KN_DEACTIVATED_BIAS); Process #1120 is trying to online memory499 by echo 1 > memory499/online In .kernfs_fop_write, it uses kernfs_get_active() to increase &kn->active, thus blocking process #1141. While itself is blocked later when trying to acquire memory_hotplug_mutex, which is held by process The backtrace of both processes are shown below: [<c000000001b18600>] 0xc000000001b18600 [<c000000000015044>] .__switch_to+0x144/0x200 [<c000000000263ca4>] .online_pages+0x74/0x7b0 [<c00000000055b40c>] .memory_subsys_online+0x9c/0x150 [<c00000000053cbe8>] .device_online+0xb8/0x120 [<c00000000053cd04>] .online_store+0xb4/0xc0 [<c000000000538ce4>] .dev_attr_store+0x64/0xa0 [<c00000000030f4ec>] .sysfs_kf_write+0x7c/0xb0 [<c00000000030e574>] .kernfs_fop_write+0x154/0x1e0 [<c000000000268450>] .vfs_write+0xe0/0x260 [<c000000000269144>] .SyS_write+0x64/0x110 [<c000000000009ffc>] syscall_exit+0x0/0x7c [<c000000001b18600>] 0xc000000001b18600 [<c000000000015044>] .__switch_to+0x144/0x200 [<c00000000030be14>] .__kernfs_remove+0x204/0x300 [<c00000000030d428>] .kernfs_remove_by_name_ns+0x68/0xf0 [<c00000000030fb38>] .sysfs_remove_file_ns+0x38/0x60 [<c000000000539354>] .device_remove_attrs+0x54/0xc0 [<c000000000539fd8>] .device_del+0x158/0x250 [<c00000000053a104>] .device_unregister+0x34/0xa0 [<c00000000055bc14>] .unregister_memory_section+0x164/0x170 [<c00000000024ee18>] .__remove_pages+0x108/0x4c0 [<c00000000004b590>] .arch_remove_memory+0x60/0xc0 [<c00000000026446c>] .remove_memory+0x8c/0xe0 [<c00000000007f9f4>] .pseries_remove_memblock+0xd4/0x160 [<c00000000007fcfc>] .pseries_memory_notifier+0x27c/0x290 [<c0000000008ae6cc>] .notifier_call_chain+0x8c/0x100 [<c0000000000d858c>] .__blocking_notifier_call_chain+0x6c/0xe0 [<c00000000071ddec>] .of_property_notify+0x7c/0xc0 [<c00000000071ed3c>] .of_update_property+0x3c/0x1b0 [<c0000000000756cc>] .ofdt_write+0x3dc/0x740 [<c0000000002f60fc>] .proc_reg_write+0xac/0x110 [<c000000000268450>] .vfs_write+0xe0/0x260 [<c000000000269144>] .SyS_write+0x64/0x110 [<c000000000009ffc>] syscall_exit+0x0/0x7c This patch uses lock_device_hotplug() to protect remove_memory() called in pseries_remove_memblock(), which is also stated before function remove_memory(): * NOTE: The caller must call lock_device_hotplug() to serialize hotplug * and online/offline operations before this call, as required by * try_offline_node(). */ void __ref remove_memory(int nid, u64 start, u64 size) With this lock held, the other process(#1120 above) trying to online the memory block will retry the system call when calling lock_device_hotplug_sysfs(), and finally find No such device error. Signed-off-by:Li Zhong <zhong@linux.vnet.ibm.com> Signed-off-by:
-
Anton Blanchard authored
module_init should return 0 or a negative errno. Signed-off-by:
Anton Blanchard <anton@samba.org> Signed-off-by:
-
Anton Blanchard authored
Bump the boot wrapper BOOT_COMMAND_LINE_SIZE to match the kernel. Signed-off-by:
-
Anton Blanchard authored
I've had a report that the current limit is too small for an automated network based installer. Bump it. Signed-off-by:
-
Anton Blanchard authored
We have two definitions of COMMAND_LINE_SIZE, one for the kernel and one for the boot wrapper. I assume this is so the boot wrapper can be self sufficient and not rely on kernel headers. Having two defines with the same name is confusing, I just updated the wrong one when trying to bump it. Make the boot wrapper define unique by calling it BOOT_COMMAND_LINE_SIZE. Signed-off-by:
-
Cody P Schafer authored
The catalog version number was changed from a be32 (with proceeding 32bits of padding) to a be64, update the code to treat it as a be64 Signed-off-by:
Cody P Schafer <cody@linux.vnet.ibm.com> Signed-off-by:
-
Cody P Schafer authored
Signed-off-by:
-
Cody P Schafer authored
Signed-off-by:
-
Cody P Schafer authored
Signed-off-by:
-
Cody P Schafer authored
fixup for "powerpc/perf: Add support for the hv gpci (get performance counter info) interface". Makes the "not enabled" message less awful (and hidden unless debugging). Signed-off-by:
-
Cody P Schafer authored
fixup for "powerpc/perf: Add support for the hv 24x7 interface" Makes the "not enabled" message less awful (and hides it in most cases). Signed-off-by:
-
Aneesh Kumar K.V authored
The if condition check was based on a draft ISA doc. Remove the same. Signed-off-by:
Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com> Signed-off-by:
-
Anton Blanchard authored
Signed-off-by:
-
Anton Blanchard authored
We have two copies of code that creates an OPAL sg list. Consolidate these into a common set of helpers and fix the endian issues. The flash interface embedded a version number in the num_entries field, whereas the dump interface did did not. Since versioning wasn't added to the flash interface and it is impossible to add this in a backwards compatible way, just remove it. Signed-off-by:
-
Anton Blanchard authored
Fix little endian issues with the OPAL error log code. Signed-off-by:
Stewart Smith <stewart@linux.vnet.ibm.com> Signed-off-by:
-
Anton Blanchard authored
The bitmap in opal_poll_events and opal_handle_interrupt is big endian, so we need to byteswap it on little endian builds. Signed-off-by:
-
Anton Blanchard authored
We had some duplication of the internal OPAL functions. Signed-off-by:
-
Anton Blanchard authored
Using size_t in our APIs is asking for trouble, especially when some OPAL calls use size_t pointers. Signed-off-by:
-
Wei Yang authored
On PowerNV platform, we are holding an unnecessary refcount on a pci_dev, which leads to the pci_dev is not destroyed when hotplugging a pci device. This patch release the unnecessary refcount. Signed-off-by:
Wei Yang <weiyang@linux.vnet.ibm.com> Signed-off-by:
-
Wei Yang authored
During the EEH hotplug event, iommu_add_device() will be invoked three times and two of them will trigger warning or error. The three times to invoke the iommu_add_device() are: pci_device_add ... set_iommu_table_base_and_group <- 1st time, fail device_add ... tce_iommu_bus_notifier <- 2nd time, succees pcibios_add_pci_devices ... pcibios_setup_bus_devices <- 3rd time, re-attach The first time fails, since the dev->kobj->sd is not initialized. The dev->kobj->sd is initialized in device_add(). The third time's warning is triggered by the re-attach of the iommu_group. After applying this patch, the error iommu_tce: 0003:05:00.0 has not been added, ret=-14 and the warning [ 204.123609] ------------[ cut here ]------------ [ 204.123645] WARNING: at arch/powerpc/kernel/iommu.c:1125 [ 204.123680] Modules linked in: xt_CHECKSUM nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_MASQUERADE ip6t_REJECT bnep bluetooth 6lowpan_iphc rfkill xt_conntrack ebtable_nat ebtable_broute bridge stp llc mlx4_ib ib_sa ib_mad ib_core ib_addr ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw bnx2x tg3 mlx4_core nfsd ptp mdio ses libcrc32c nfs_acl enclosure be2net pps_core shpchp lockd kvm uinput sunrpc binfmt_misc lpfc scsi_transport_fc ipr scsi_tgt [ 204.124356] CPU: 18 PID: 650 Comm: eehd Not tainted 3.14.0-rc5yw+ #102 [ 204.124400] task: c0000027ed485670 ti: c0000027ed50c000 task.ti: c0000027ed50c000 [ 204.124453] NIP: c00000000003cf80 LR: c00000000006c648 CTR: c00000000006c5c0 [ 204.124506] REGS: c0000027ed50f440 TRAP: 0700 Not tainted (3.14.0-rc5yw+) [ 204.124558] MSR: 9000000000029032 <SF,HV,EE,ME,IR,DR,RI> CR: 88008084 XER: 20000000 [ 204.124682] CFAR: c00000000006c644 SOFTE: 1 GPR00: c00000000006c648 c0000027ed50f6c0 c000000001398380 c0000027ec260300 GPR04: c0000027ea92c000 c00000000006ad00 c0000000016e41b0 0000000000000110 GPR08: c0000000012cd4c0 0000000000000001 c0000027ec2602ff 0000000000000062 GPR12: 0000000028008084 c00000000fdca200 c0000000000d1d90 c0000027ec281a80 GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 GPR20: 0000000000000000 0000000000000000 0000000000000000 0000000000000001 GPR24: 000000005342697b 0000000000002906 c000001fe6ac9800 c000001fe6ac9800 GPR28: 0000000000000000 c0000000016e3a80 c0000027ea92c090 c0000027ea92c000 [ 204.125353] NIP [c00000000003cf80] .iommu_add_device+0x30/0x1f0 [ 204.125399] LR [c00000000006c648] .pnv_pci_ioda_dma_dev_setup+0x88/0xb0 [ 204.125443] Call Trace: [ 204.125464] [c0000027ed50f6c0] [c0000027ed50f750] 0xc0000027ed50f750 (unreliable) [ 204.125526] [c0000027ed50f750] [c00000000006c648] .pnv_pci_ioda_dma_dev_setup+0x88/0xb0 [ 204.125588] [c0000027ed50f7d0] [c000000000069cc8] .pnv_pci_dma_dev_setup+0x78/0x340 [ 204.125650] [c0000027ed50f870] [c000000000044408] .pcibios_setup_device+0x88/0x2f0 [ 204.125712] [c0000027ed50f940] [c000000000046040] .pcibios_setup_bus_devices+0x60/0xd0 [ 204.125774] [c0000027ed50f9c0] [c000000000043acc] .pcibios_add_pci_devices+0xdc/0x1c0 [ 204.125837] [c0000027ed50fa50] [c00000000086f970] .eeh_reset_device+0x36c/0x4f0 [ 204.125939] [c0000027ed50fb20] [c00000000003a2d8] .eeh_handle_normal_event+0x448/0x480 [ 204.126068] [c0000027ed50fbc0] [c00000000003a35c] .eeh_handle_event+0x4c/0x340 [ 204.126192] [c0000027ed50fc80] [c00000000003a74c] .eeh_event_handler+0xfc/0x1b0 [ 204.126319] [c0000027ed50fd30] [c0000000000d1ea0] .kthread+0x110/0x130 [ 204.126430] [c0000027ed50fe30] [c00000000000a460] .ret_from_kernel_thread+0x5c/0x7c [ 204.126556] Instruction dump: [ 204.126610] 7c0802a6 fba1ffe8 fbc1fff0 fbe1fff8 f8010010 f821ff71 7c7e1b78 60000000 [ 204.126787] 60000000 e87e0298 3143ffff 7d2a1910 <0b090000> 2fa90000 40de00c8 ebfe0218 [ 204.126966] ---[ end trace 6e7aefd80add2973 ]--- are cleared. This patch removes iommu_add_device() in pnv_pci_ioda_dma_dev_setup(), which revert part of the change in commit d905c5df(PPC: POWERNV: move iommu_add_device earlier). Signed-off-by: -
Anton Blanchard authored
With this patch I was able to update firmware on an LE kernel. Signed-off-by:
-
Benjamin Herrenschmidt authored
We have a subtle race when sending CPUs back to OPAL on kexec. We mark them as "in real mode" right before we send them down. Once we've booted the new kernel, it might try to call opal_reinit_cpus() to change endianness, and that requires all CPUs to be spinning inside OPAL. However there is no synchronization here and we've observed cases where the returning CPUs hadn't established their new state inside OPAL before opal_reinit_cpus() is called, causing it to fail. The proper fix is to actually wait for them to go down all the way from the kexec'ing kernel. Signed-off-by: -
Joel Stanley authored
The size of the sysparam sysfs files is determined from the device tree at boot. However the buffer is hard coded to 64 bytes. If we encounter a parameter that is larger than 64, or miss-parse the device tree, the buffer will overflow when reading or writing to the parameter. Check it at discovery time, and if the parameter is too large, do not create a sysfs entry for it. Signed-off-by:
Joel Stanley <joel@jms.id.au> Signed-off-by:
-
