Commit e6d69a60 authored by Linus Torvalds's avatar Linus Torvalds

Merge branch 'next' of git://git.infradead.org/users/vkoul/slave-dma

Pull slave-dmaengine changes from Vinod Koul:
 "This brings for slave dmaengine:

   - Change dma notification flag to DMA_COMPLETE from DMA_SUCCESS as
     dmaengine can only transfer and not verify validaty of dma
     transfers

   - Bunch of fixes across drivers:

      - cppi41 driver fixes from Daniel

      - 8 channel freescale dma engine support and updated bindings from
        Hongbo

      - msx-dma fixes and cleanup by Markus

   - DMAengine updates from Dan:

      - Bartlomiej and Dan finalized a rework of the dma address unmap
        implementation.

      - In the course of testing 1/ a collection of enhancements to
        dmatest fell out.  Notably basic performance statistics, and
        fixed / enhanced test control through new module parameters
        'run', 'wait', 'noverify', and 'verbose'.  Thanks to Andriy and
        Linus [Walleij] for their review.

      - Testing the raid related corner cases of 1/ triggered bugs in
        the recently added 16-source operation support in the ioatdma
        driver.

      - Some minor fixes / cleanups to mv_xor and ioatdma"

* 'next' of git://git.infradead.org/users/vkoul/slave-dma: (99 commits)
  dma: mv_xor: Fix mis-usage of mmio 'base' and 'high_base' registers
  dma: mv_xor: Remove unneeded NULL address check
  ioat: fix ioat3_irq_reinit
  ioat: kill msix_single_vector support
  raid6test: add new corner case for ioatdma driver
  ioatdma: clean up sed pool kmem_cache
  ioatdma: fix selection of 16 vs 8 source path
  ioatdma: fix sed pool selection
  ioatdma: Fix bug in selftest after removal of DMA_MEMSET.
  dmatest: verbose mode
  dmatest: convert to dmaengine_unmap_data
  dmatest: add a 'wait' parameter
  dmatest: add basic performance metrics
  dmatest: add support for skipping verification and random data setup
  dmatest: use pseudo random numbers
  dmatest: support xor-only, or pq-only channels in tests
  dmatest: restore ability to start test at module load and init
  dmatest: cleanup redundant "dmatest: " prefixes
  dmatest: replace stored results mechanism, with uniform messages
  Revert "dmatest: append verify result to results"
  ...
parents 5a1efc6e df12a317
...@@ -28,7 +28,7 @@ The three cells in order are: ...@@ -28,7 +28,7 @@ The three cells in order are:
dependent: dependent:
- bit 7-0: peripheral identifier for the hardware handshaking interface. The - bit 7-0: peripheral identifier for the hardware handshaking interface. The
identifier can be different for tx and rx. identifier can be different for tx and rx.
- bit 11-8: FIFO configuration. 0 for half FIFO, 1 for ALAP, 1 for ASAP. - bit 11-8: FIFO configuration. 0 for half FIFO, 1 for ALAP, 2 for ASAP.
Example: Example:
......
* Freescale 83xx DMA Controller * Freescale DMA Controllers
Freescale PowerPC 83xx have on chip general purpose DMA controllers. ** Freescale Elo DMA Controller
This is a little-endian 4-channel DMA controller, used in Freescale mpc83xx
series chips such as mpc8315, mpc8349, mpc8379 etc.
Required properties: Required properties:
- compatible : compatible list, contains 2 entries, first is - compatible : must include "fsl,elo-dma"
"fsl,CHIP-dma", where CHIP is the processor - reg : DMA General Status Register, i.e. DGSR which contains
(mpc8349, mpc8360, etc.) and the second is status for all the 4 DMA channels
"fsl,elo-dma" - ranges : describes the mapping between the address space of the
- reg : <registers mapping for DMA general status reg> DMA channels and the address space of the DMA controller
- ranges : Should be defined as specified in 1) to describe the
DMA controller channels.
- cell-index : controller index. 0 for controller @ 0x8100 - cell-index : controller index. 0 for controller @ 0x8100
- interrupts : <interrupt mapping for DMA IRQ> - interrupts : interrupt specifier for DMA IRQ
- interrupt-parent : optional, if needed for interrupt mapping - interrupt-parent : optional, if needed for interrupt mapping
- DMA channel nodes: - DMA channel nodes:
- compatible : compatible list, contains 2 entries, first is - compatible : must include "fsl,elo-dma-channel"
"fsl,CHIP-dma-channel", where CHIP is the processor However, see note below.
(mpc8349, mpc8350, etc.) and the second is - reg : DMA channel specific registers
"fsl,elo-dma-channel". However, see note below. - cell-index : DMA channel index starts at 0.
- reg : <registers mapping for channel>
- cell-index : dma channel index starts at 0.
Optional properties: Optional properties:
- interrupts : <interrupt mapping for DMA channel IRQ> - interrupts : interrupt specifier for DMA channel IRQ
(on 83xx this is expected to be identical to (on 83xx this is expected to be identical to
the interrupts property of the parent node) the interrupts property of the parent node)
- interrupt-parent : optional, if needed for interrupt mapping - interrupt-parent : optional, if needed for interrupt mapping
Example: Example:
...@@ -70,30 +67,27 @@ Example: ...@@ -70,30 +67,27 @@ Example:
}; };
}; };
* Freescale 85xx/86xx DMA Controller ** Freescale EloPlus DMA Controller
This is a 4-channel DMA controller with extended addresses and chaining,
Freescale PowerPC 85xx/86xx have on chip general purpose DMA controllers. mainly used in Freescale mpc85xx/86xx, Pxxx and BSC series chips, such as
mpc8540, mpc8641 p4080, bsc9131 etc.
Required properties: Required properties:
- compatible : compatible list, contains 2 entries, first is - compatible : must include "fsl,eloplus-dma"
"fsl,CHIP-dma", where CHIP is the processor - reg : DMA General Status Register, i.e. DGSR which contains
(mpc8540, mpc8540, etc.) and the second is status for all the 4 DMA channels
"fsl,eloplus-dma"
- reg : <registers mapping for DMA general status reg>
- cell-index : controller index. 0 for controller @ 0x21000, - cell-index : controller index. 0 for controller @ 0x21000,
1 for controller @ 0xc000 1 for controller @ 0xc000
- ranges : Should be defined as specified in 1) to describe the - ranges : describes the mapping between the address space of the
DMA controller channels. DMA channels and the address space of the DMA controller
- DMA channel nodes: - DMA channel nodes:
- compatible : compatible list, contains 2 entries, first is - compatible : must include "fsl,eloplus-dma-channel"
"fsl,CHIP-dma-channel", where CHIP is the processor However, see note below.
(mpc8540, mpc8560, etc.) and the second is - cell-index : DMA channel index starts at 0.
"fsl,eloplus-dma-channel". However, see note below. - reg : DMA channel specific registers
- cell-index : dma channel index starts at 0. - interrupts : interrupt specifier for DMA channel IRQ
- reg : <registers mapping for channel>
- interrupts : <interrupt mapping for DMA channel IRQ>
- interrupt-parent : optional, if needed for interrupt mapping - interrupt-parent : optional, if needed for interrupt mapping
Example: Example:
...@@ -134,6 +128,76 @@ Example: ...@@ -134,6 +128,76 @@ Example:
}; };
}; };
** Freescale Elo3 DMA Controller
DMA controller which has same function as EloPlus except that Elo3 has 8
channels while EloPlus has only 4, it is used in Freescale Txxx and Bxxx
series chips, such as t1040, t4240, b4860.
Required properties:
- compatible : must include "fsl,elo3-dma"
- reg : contains two entries for DMA General Status Registers,
i.e. DGSR0 which includes status for channel 1~4, and
DGSR1 for channel 5~8
- ranges : describes the mapping between the address space of the
DMA channels and the address space of the DMA controller
- DMA channel nodes:
- compatible : must include "fsl,eloplus-dma-channel"
- reg : DMA channel specific registers
- interrupts : interrupt specifier for DMA channel IRQ
- interrupt-parent : optional, if needed for interrupt mapping
Example:
dma@100300 {
#address-cells = <1>;
#size-cells = <1>;
compatible = "fsl,elo3-dma";
reg = <0x100300 0x4>,
<0x100600 0x4>;
ranges = <0x0 0x100100 0x500>;
dma-channel@0 {
compatible = "fsl,eloplus-dma-channel";
reg = <0x0 0x80>;
interrupts = <28 2 0 0>;
};
dma-channel@80 {
compatible = "fsl,eloplus-dma-channel";
reg = <0x80 0x80>;
interrupts = <29 2 0 0>;
};
dma-channel@100 {
compatible = "fsl,eloplus-dma-channel";
reg = <0x100 0x80>;
interrupts = <30 2 0 0>;
};
dma-channel@180 {
compatible = "fsl,eloplus-dma-channel";
reg = <0x180 0x80>;
interrupts = <31 2 0 0>;
};
dma-channel@300 {
compatible = "fsl,eloplus-dma-channel";
reg = <0x300 0x80>;
interrupts = <76 2 0 0>;
};
dma-channel@380 {
compatible = "fsl,eloplus-dma-channel";
reg = <0x380 0x80>;
interrupts = <77 2 0 0>;
};
dma-channel@400 {
compatible = "fsl,eloplus-dma-channel";
reg = <0x400 0x80>;
interrupts = <78 2 0 0>;
};
dma-channel@480 {
compatible = "fsl,eloplus-dma-channel";
reg = <0x480 0x80>;
interrupts = <79 2 0 0>;
};
};
Note on DMA channel compatible properties: The compatible property must say Note on DMA channel compatible properties: The compatible property must say
"fsl,elo-dma-channel" or "fsl,eloplus-dma-channel" to be used by the Elo DMA "fsl,elo-dma-channel" or "fsl,eloplus-dma-channel" to be used by the Elo DMA
driver (fsldma). Any DMA channel used by fsldma cannot be used by another driver (fsldma). Any DMA channel used by fsldma cannot be used by another
......
...@@ -15,39 +15,48 @@ be built as module or inside kernel. Let's consider those cases. ...@@ -15,39 +15,48 @@ be built as module or inside kernel. Let's consider those cases.
Part 2 - When dmatest is built as a module... Part 2 - When dmatest is built as a module...
After mounting debugfs and loading the module, the /sys/kernel/debug/dmatest
folder with nodes will be created. There are two important files located. First
is the 'run' node that controls run and stop phases of the test, and the second
one, 'results', is used to get the test case results.
Note that in this case test will not run on load automatically.
Example of usage: Example of usage:
% modprobe dmatest channel=dma0chan0 timeout=2000 iterations=1 run=1
...or:
% modprobe dmatest
% echo dma0chan0 > /sys/module/dmatest/parameters/channel % echo dma0chan0 > /sys/module/dmatest/parameters/channel
% echo 2000 > /sys/module/dmatest/parameters/timeout % echo 2000 > /sys/module/dmatest/parameters/timeout
% echo 1 > /sys/module/dmatest/parameters/iterations % echo 1 > /sys/module/dmatest/parameters/iterations
% echo 1 > /sys/kernel/debug/dmatest/run % echo 1 > /sys/module/dmatest/parameters/run
...or on the kernel command line:
dmatest.channel=dma0chan0 dmatest.timeout=2000 dmatest.iterations=1 dmatest.run=1
Hint: available channel list could be extracted by running the following Hint: available channel list could be extracted by running the following
command: command:
% ls -1 /sys/class/dma/ % ls -1 /sys/class/dma/
After a while you will start to get messages about current status or error like Once started a message like "dmatest: Started 1 threads using dma0chan0" is
in the original code. emitted. After that only test failure messages are reported until the test
stops.
Note that running a new test will not stop any in progress test. Note that running a new test will not stop any in progress test.
The following command should return actual state of the test. The following command returns the state of the test.
% cat /sys/kernel/debug/dmatest/run % cat /sys/module/dmatest/parameters/run
To wait for test done the user may perform a busy loop that checks the state. To wait for test completion userpace can poll 'run' until it is false, or use
the wait parameter. Specifying 'wait=1' when loading the module causes module
% while [ $(cat /sys/kernel/debug/dmatest/run) = "Y" ] initialization to pause until a test run has completed, while reading
> do /sys/module/dmatest/parameters/wait waits for any running test to complete
> echo -n "." before returning. For example, the following scripts wait for 42 tests
> sleep 1 to complete before exiting. Note that if 'iterations' is set to 'infinite' then
> done waiting is disabled.
> echo
Example:
% modprobe dmatest run=1 iterations=42 wait=1
% modprobe -r dmatest
...or:
% modprobe dmatest run=1 iterations=42
% cat /sys/module/dmatest/parameters/wait
% modprobe -r dmatest
Part 3 - When built-in in the kernel... Part 3 - When built-in in the kernel...
...@@ -62,21 +71,22 @@ case. You always could check them at run-time by running ...@@ -62,21 +71,22 @@ case. You always could check them at run-time by running
Part 4 - Gathering the test results Part 4 - Gathering the test results
The module provides a storage for the test results in the memory. The gathered Test results are printed to the kernel log buffer with the format:
data could be used after test is done.
The special file 'results' in the debugfs represents gathered data of the in "dmatest: result <channel>: <test id>: '<error msg>' with src_off=<val> dst_off=<val> len=<val> (<err code>)"
progress test. The messages collected are printed to the kernel log as well.
Example of output: Example of output:
% cat /sys/kernel/debug/dmatest/results % dmesg | tail -n 1
dma0chan0-copy0: #1: No errors with src_off=0x7bf dst_off=0x8ad len=0x3fea (0) dmatest: result dma0chan0-copy0: #1: No errors with src_off=0x7bf dst_off=0x8ad len=0x3fea (0)
The message format is unified across the different types of errors. A number in The message format is unified across the different types of errors. A number in
the parens represents additional information, e.g. error code, error counter, the parens represents additional information, e.g. error code, error counter,
or status. or status. A test thread also emits a summary line at completion listing the
number of tests executed, number that failed, and a result code.
Comparison between buffers is stored to the dedicated structure. Example:
% dmesg | tail -n 1
dmatest: dma0chan0-copy0: summary 1 test, 0 failures 1000 iops 100000 KB/s (0)
Note that the verify result is now accessible only via file 'results' in the The details of a data miscompare error are also emitted, but do not follow the
debugfs. above format.
...@@ -404,7 +404,7 @@ static irqreturn_t dma_irq_handler(int irq, void *data) ...@@ -404,7 +404,7 @@ static irqreturn_t dma_irq_handler(int irq, void *data)
BIT(slot)); BIT(slot));
if (edma_cc[ctlr]->intr_data[channel].callback) if (edma_cc[ctlr]->intr_data[channel].callback)
edma_cc[ctlr]->intr_data[channel].callback( edma_cc[ctlr]->intr_data[channel].callback(
channel, DMA_COMPLETE, channel, EDMA_DMA_COMPLETE,
edma_cc[ctlr]->intr_data[channel].data); edma_cc[ctlr]->intr_data[channel].data);
} }
} while (sh_ipr); } while (sh_ipr);
...@@ -459,7 +459,7 @@ static irqreturn_t dma_ccerr_handler(int irq, void *data) ...@@ -459,7 +459,7 @@ static irqreturn_t dma_ccerr_handler(int irq, void *data)
callback) { callback) {
edma_cc[ctlr]->intr_data[k]. edma_cc[ctlr]->intr_data[k].
callback(k, callback(k,
DMA_CC_ERROR, EDMA_DMA_CC_ERROR,
edma_cc[ctlr]->intr_data edma_cc[ctlr]->intr_data
[k].data); [k].data);
} }
......
...@@ -393,36 +393,6 @@ static inline int iop_chan_zero_sum_slot_count(size_t len, int src_cnt, ...@@ -393,36 +393,6 @@ static inline int iop_chan_zero_sum_slot_count(size_t len, int src_cnt,
return slot_cnt; return slot_cnt;
} }
static inline int iop_desc_is_pq(struct iop_adma_desc_slot *desc)
{
return 0;
}
static inline u32 iop_desc_get_dest_addr(struct iop_adma_desc_slot *desc,
struct iop_adma_chan *chan)
{
union iop3xx_desc hw_desc = { .ptr = desc->hw_desc, };
switch (chan->device->id) {
case DMA0_ID:
case DMA1_ID:
return hw_desc.dma->dest_addr;
case AAU_ID:
return hw_desc.aau->dest_addr;
default:
BUG();
}
return 0;
}
static inline u32 iop_desc_get_qdest_addr(struct iop_adma_desc_slot *desc,
struct iop_adma_chan *chan)
{
BUG();
return 0;
}
static inline u32 iop_desc_get_byte_count(struct iop_adma_desc_slot *desc, static inline u32 iop_desc_get_byte_count(struct iop_adma_desc_slot *desc,
struct iop_adma_chan *chan) struct iop_adma_chan *chan)
{ {
......
...@@ -82,8 +82,6 @@ struct iop_adma_chan { ...@@ -82,8 +82,6 @@ struct iop_adma_chan {
* @slot_cnt: total slots used in an transaction (group of operations) * @slot_cnt: total slots used in an transaction (group of operations)
* @slots_per_op: number of slots per operation * @slots_per_op: number of slots per operation
* @idx: pool index * @idx: pool index
* @unmap_src_cnt: number of xor sources
* @unmap_len: transaction bytecount
* @tx_list: list of descriptors that are associated with one operation * @tx_list: list of descriptors that are associated with one operation
* @async_tx: support for the async_tx api * @async_tx: support for the async_tx api
* @group_list: list of slots that make up a multi-descriptor transaction * @group_list: list of slots that make up a multi-descriptor transaction
...@@ -99,8 +97,6 @@ struct iop_adma_desc_slot { ...@@ -99,8 +97,6 @@ struct iop_adma_desc_slot {
u16 slot_cnt; u16 slot_cnt;
u16 slots_per_op; u16 slots_per_op;
u16 idx; u16 idx;
u16 unmap_src_cnt;
size_t unmap_len;
struct list_head tx_list; struct list_head tx_list;
struct dma_async_tx_descriptor async_tx; struct dma_async_tx_descriptor async_tx;
union { union {
......
...@@ -218,20 +218,6 @@ iop_chan_xor_slot_count(size_t len, int src_cnt, int *slots_per_op) ...@@ -218,20 +218,6 @@ iop_chan_xor_slot_count(size_t len, int src_cnt, int *slots_per_op)
#define iop_chan_pq_slot_count iop_chan_xor_slot_count #define iop_chan_pq_slot_count iop_chan_xor_slot_count
#define iop_chan_pq_zero_sum_slot_count iop_chan_xor_slot_count #define iop_chan_pq_zero_sum_slot_count iop_chan_xor_slot_count
static inline u32 iop_desc_get_dest_addr(struct iop_adma_desc_slot *desc,
struct iop_adma_chan *chan)
{
struct iop13xx_adma_desc_hw *hw_desc = desc->hw_desc;
return hw_desc->dest_addr;
}
static inline u32 iop_desc_get_qdest_addr(struct iop_adma_desc_slot *desc,
struct iop_adma_chan *chan)
{
struct iop13xx_adma_desc_hw *hw_desc = desc->hw_desc;
return hw_desc->q_dest_addr;
}
static inline u32 iop_desc_get_byte_count(struct iop_adma_desc_slot *desc, static inline u32 iop_desc_get_byte_count(struct iop_adma_desc_slot *desc,
struct iop_adma_chan *chan) struct iop_adma_chan *chan)
{ {
...@@ -350,18 +336,6 @@ iop_desc_init_pq(struct iop_adma_desc_slot *desc, int src_cnt, ...@@ -350,18 +336,6 @@ iop_desc_init_pq(struct iop_adma_desc_slot *desc, int src_cnt,
hw_desc->desc_ctrl = u_desc_ctrl.value; hw_desc->desc_ctrl = u_desc_ctrl.value;
} }
static inline int iop_desc_is_pq(struct iop_adma_desc_slot *desc)
{
struct iop13xx_adma_desc_hw *hw_desc = desc->hw_desc;
union {
u32 value;
struct iop13xx_adma_desc_ctrl field;
} u_desc_ctrl;
u_desc_ctrl.value = hw_desc->desc_ctrl;
return u_desc_ctrl.field.pq_xfer_en;
}
static inline void static inline void
iop_desc_init_pq_zero_sum(struct iop_adma_desc_slot *desc, int src_cnt, iop_desc_init_pq_zero_sum(struct iop_adma_desc_slot *desc, int src_cnt,
unsigned long flags) unsigned long flags)
......
...@@ -223,13 +223,13 @@ rcpm: global-utilities@e2000 { ...@@ -223,13 +223,13 @@ rcpm: global-utilities@e2000 {
reg = <0xe2000 0x1000>; reg = <0xe2000 0x1000>;
}; };
/include/ "qoriq-dma-0.dtsi" /include/ "elo3-dma-0.dtsi"
dma@100300 { dma@100300 {
fsl,iommu-parent = <&pamu0>; fsl,iommu-parent = <&pamu0>;
fsl,liodn-reg = <&guts 0x580>; /* DMA1LIODNR */ fsl,liodn-reg = <&guts 0x580>; /* DMA1LIODNR */
}; };
/include/ "qoriq-dma-1.dtsi" /include/ "elo3-dma-1.dtsi"
dma@101300 { dma@101300 {
fsl,iommu-parent = <&pamu0>; fsl,iommu-parent = <&pamu0>;
fsl,liodn-reg = <&guts 0x584>; /* DMA2LIODNR */ fsl,liodn-reg = <&guts 0x584>; /* DMA2LIODNR */
......
/*
* QorIQ Elo3 DMA device tree stub [ controller @ offset 0x100000 ]
*
* Copyright 2013 Freescale Semiconductor Inc.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
* * Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Freescale Semiconductor nor the
* names of its contributors may be used to endorse or promote products
* derived from this software without specific prior written permission.
*
*
* ALTERNATIVELY, this software may be distributed under the terms of the
* GNU General Public License ("GPL") as published by the Free Software
* Foundation, either version 2 of that License or (at your option) any
* later version.
*
* THIS SOFTWARE IS PROVIDED BY Freescale Semiconductor ``AS IS'' AND ANY
* EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
* WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
* DISCLAIMED. IN NO EVENT SHALL Freescale Semiconductor BE LIABLE FOR ANY
* DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
* (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
* LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
* ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
* SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
dma0: dma@100300 {
#address-cells = <1>;
#size-cells = <1>;
compatible = "fsl,elo3-dma";
reg = <0x100300 0x4>,
<0x100600 0x4>;
ranges = <0x0 0x100100 0x500>;
dma-channel@0 {
compatible = "fsl,eloplus-dma-channel";
reg = <0x0 0x80>;
interrupts = <28 2 0 0>;
};
dma-channel@80 {
compatible = "fsl,eloplus-dma-channel";
reg = <0x80 0x80>;
interrupts = <29 2 0 0>;
};
dma-channel@100 {
compatible = "fsl,eloplus-dma-channel";
reg = <0x100 0x80>;
interrupts = <30 2 0 0>;
};
dma-channel@180 {
compatible = "fsl,eloplus-dma-channel";
reg = <0x180 0x80>;
interrupts = <31 2 0 0>;
};
dma-channel@300 {
compatible = "fsl,eloplus-dma-channel";
reg = <0x300 0x80>;
interrupts = <76 2 0 0>;
};
dma-channel@380 {
compatible = "fsl,eloplus-dma-channel";
reg = <0x380 0x80>;
interrupts = <77 2 0 0>;
};
dma-channel@400 {
compatible = "fsl,eloplus-dma-channel";
reg = <0x400 0x80>;
interrupts = <78 2 0 0>;
};
dma-channel@480 {
compatible = "fsl,eloplus-dma-channel";
reg = <0x480 0x80>;
interrupts = <79 2 0 0>;
};
};
/*
* QorIQ Elo3 DMA device tree stub [ controller @ offset 0x101000 ]
*
* Copyright 2013 Freescale Semiconductor Inc.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
* * Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Freescale Semiconductor nor the
* names of its contributors may be used to endorse or promote products
* derived from this software without specific prior written permission.
*
*
* ALTERNATIVELY, this software may be distributed under the terms of the
* GNU General Public License ("GPL") as published by the Free Software
* Foundation, either version 2 of that License or (at your option) any
* later version.
*
* THIS SOFTWARE IS PROVIDED BY Freescale Semiconductor ``AS IS'' AND ANY
* EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
* WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
* DISCLAIMED. IN NO EVENT SHALL Freescale Semiconductor BE LIABLE FOR ANY
* DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
* (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
* LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
* ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
* SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
dma1: dma@101300 {
#address-cells = <1>;
#size-cells = <1>;
compatible = "fsl,elo3-dma";
reg = <0x101300 0x4>,
<0x101600 0x4>;
ranges = <0x0 0x101100 0x500>;
dma-channel@0 {
compatible = "fsl,eloplus-dma-channel";
reg = <0x0 0x80>;
interrupts = <32 2 0 0>;
};
dma-channel@80 {
compatible = "fsl,eloplus-dma-channel";
reg = <0x80 0x80>;
interrupts = <33 2 0 0>;
};
dma-channel@100 {
compatible = "fsl,eloplus-dma-channel";
reg = <0x100 0x80>;
interrupts = <34 2 0 0>;
};
dma-channel@180 {
compatible = "fsl,eloplus-dma-channel";
reg = <0x180 0x80>;
interrupts = <35 2 0 0>;
};
dma-channel@300 {
compatible = "fsl,eloplus-dma-channel";
reg = <0x300 0x80>;
interrupts = <80 2 0 0>;
};
dma-channel@380 {
compatible = "fsl,eloplus-dma-channel";
reg = <0x380 0x80>;
interrupts = <81 2 0 0>;
};
dma-channel@400 {
compatible = "fsl,eloplus-dma-channel";
reg = <0x400 0x80>;
interrupts = <82 2 0 0>;
};
dma-channel@480 {
compatible = "fsl,eloplus-dma-channel";
reg = <0x480 0x80>;
interrupts = <83 2 0 0>;
};
};
...@@ -387,8 +387,8 @@ serdes: serdes@ea000 { ...@@ -387,8 +387,8 @@ serdes: serdes@ea000 {
reg = <0xea000 0x4000>; reg = <0xea000 0x4000>;
}; };
/include/ "qoriq-dma-0.dtsi" /include/ "elo3-dma-0.dtsi"
/include/ "qoriq-dma-1.dtsi" /include/ "elo3-dma-1.dtsi"
/include/ "qoriq-espi-0.dtsi" /include/ "qoriq-espi-0.dtsi"
spi@110000 { spi@110000 {
......
...@@ -50,33 +50,36 @@ async_memcpy(struct page *dest, struct page *src, unsigned int dest_offset, ...@@ -50,33 +50,36 @@ async_memcpy(struct page *dest, struct page *src, unsigned int dest_offset,
&dest, 1, &src, 1, len); &dest, 1, &src, 1, len);
struct dma_device *device = chan ? chan->device : NULL; struct dma_device *device = chan ? chan->device : NULL;
struct dma_async_tx_descriptor *tx = NULL; struct dma_async_tx_descriptor *tx = NULL;
struct dmaengine_unmap_data *unmap = NULL;
if (device && is_dma_copy_aligned(device, src_offset, dest_offset, len)) { if (device)
dma_addr_t dma_dest, dma_src; unmap = dmaengine_get_unmap_data(device->dev, 2, GFP_NOIO);
if (unmap && is_dma_copy_aligned(device, src_offset, dest_offset, len)) {
unsigned long dma_prep_flags = 0; unsigned long dma_prep_flags = 0;
if (submit->cb_fn) if (submit->cb_fn)
dma_prep_flags |= DMA_PREP_INTERRUPT; dma_prep_flags |= DMA_PREP_INTERRUPT;
if (submit->flags & ASYNC_TX_FENCE) if (submit->flags & ASYNC_TX_FENCE)
dma_prep_flags |= DMA_PREP_FENCE; dma_prep_flags |= DMA_PREP_FENCE;
dma_dest = dma_map_page(device->dev, dest, dest_offset, len,
DMA_FROM_DEVICE); unmap->to_cnt = 1;
unmap->addr[0] = dma_map_page(device->dev, src, src_offset, len,
dma_src = dma_map_page(device->dev, src, src_offset, len, DMA_TO_DEVICE);
DMA_TO_DEVICE); unmap->from_cnt = 1;
unmap->addr[1] = dma_map_page(device->dev, dest, dest_offset, len,
tx = device->device_prep_dma_memcpy(chan, dma_dest, dma_src, DMA_FROM_DEVICE);
len, dma_prep_flags); unmap->len = len;
if (!tx) {
dma_unmap_page(device->dev, dma_dest, len, tx = device->device_prep_dma_memcpy(chan, unmap->addr[1],
DMA_FROM_DEVICE); unmap->addr[0], len,
dma_unmap_page(device->dev, dma_src, len, dma_prep_flags);
DMA_TO_DEVICE);
}
} }
if (tx) { if (tx) {
pr_debug("%s: (async) len: %zu\n", __func__, len); pr_debug("%s: (async) len: %zu\n", __func__, len);
dma_set_unmap(tx, unmap);
async_tx_submit(chan, tx, submit); async_tx_submit(chan, tx, submit);
} else { } else {
void *dest_buf, *src_buf; void *dest_buf, *src_buf;
...@@ -96,6 +99,8 @@ async_memcpy(struct page *dest, struct page *src, unsigned int dest_offset, ...@@ -96,6 +99,8 @@ async_memcpy(struct page *dest, struct page *src, unsigned int dest_offset,
async_tx_sync_epilog(submit); async_tx_sync_epilog(submit);
} }
dmaengine_unmap_put(unmap);
return tx; return tx;
} }
EXPORT_SYMBOL_GPL(async_memcpy); EXPORT_SYMBOL_GPL(async_memcpy);
......
...@@ -46,49 +46,24 @@ static struct page *pq_scribble_page; ...@@ -46,49 +46,24 @@ static struct page *pq_scribble_page;
* do_async_gen_syndrome - asynchronously calculate P and/or Q * do_async_gen_syndrome - asynchronously calculate P and/or Q
*/ */
static __async_inline struct dma_async_tx_descriptor * static __async_inline struct dma_async_tx_descriptor *
do_async_gen_syndrome(struct dma_chan *chan, struct page **blocks, do_async_gen_syndrome(struct dma_chan *chan,
const unsigned char *scfs, unsigned int offset, int disks, const unsigned char *scfs, int disks,
size_t len, dma_addr_t *dma_src, struct dmaengine_unmap_data *unmap,
enum dma_ctrl_flags dma_flags,
struct async_submit_ctl *submit) struct async_submit_ctl *submit)
{ {
struct dma_async_tx_descriptor *tx = NULL; struct dma_async_tx_descriptor *tx = NULL;
struct dma_device *dma = chan->device; struct dma_device *dma = chan->device;
enum dma_ctrl_flags dma_flags = 0;
enum async_tx_flags flags_orig = submit->flags; enum async_tx_flags flags_orig = submit->flags;
dma_async_tx_callback cb_fn_orig = submit->cb_fn; dma_async_tx_callback cb_fn_orig = submit->cb_fn;
dma_async_tx_callback cb_param_orig = submit->cb_param; dma_async_tx_callback cb_param_orig = submit->cb_param;
int src_cnt = disks - 2; int src_cnt = disks - 2;
unsigned char coefs[src_cnt];
unsigned short pq_src_cnt; unsigned short pq_src_cnt;
dma_addr_t dma_dest[2]; dma_addr_t dma_dest[2];
int src_off = 0; int src_off = 0;
int idx;
int i;
/* DMAs use destinations as sources, so use BIDIRECTIONAL mapping */ if (submit->flags & ASYNC_TX_FENCE)
if (P(blocks, disks)) dma_flags |= DMA_PREP_FENCE;
dma_dest[0] = dma_map_page(dma->dev, P(blocks, disks), offset,
len, DMA_BIDIRECTIONAL);
else
dma_flags |= DMA_PREP_PQ_DISABLE_P;
if (Q(blocks, disks))
dma_dest[1] = dma_map_page(dma->dev, Q(blocks, disks), offset,
len, DMA_BIDIRECTIONAL);
else
dma_flags |= DMA_PREP_PQ_DISABLE_Q;
/* convert source addresses being careful to collapse 'empty'
* sources and update the coefficients accordingly
*/
for (i = 0, idx = 0; i < src_cnt; i++) {
if (blocks[i] == NULL)
continue;
dma_src[idx] = dma_map_page(dma->dev, blocks[i], offset, len,
DMA_TO_DEVICE);
coefs[idx] = scfs[i];
idx++;
}
src_cnt = idx;
while (src_cnt > 0) { while (src_cnt > 0) {
submit->flags = flags_orig; submit->flags = flags_orig;
...@@ -100,28 +75,25 @@ do_async_gen_syndrome(struct dma_chan *chan, struct page **blocks, ...@@ -100,28 +75,25 @@ do_async_gen_syndrome(struct dma_chan *chan, struct page **blocks,
if (src_cnt > pq_src_cnt) { if (src_cnt > pq_src_cnt) {
submit->flags &= ~ASYNC_TX_ACK; submit->flags &= ~ASYNC_TX_ACK;
submit->flags |= ASYNC_TX_FENCE; submit->flags |= ASYNC_TX_FENCE;
dma_flags |= DMA_COMPL_SKIP_DEST_UNMAP;
submit->cb_fn = NULL; submit->cb_fn = NULL;
submit->cb_param = NULL; submit->cb_param = NULL;
} else { } else {
dma_flags &= ~DMA_COMPL_SKIP_DEST_UNMAP;
submit->cb_fn = cb_fn_orig; submit->cb_fn = cb_fn_orig;
submit->cb_param = cb_param_orig; submit->cb_param = cb_param_orig;
if (cb_fn_orig) if (cb_fn_orig)
dma_flags |= DMA_PREP_INTERRUPT; dma_flags |= DMA_PREP_INTERRUPT;
} }
if (submit->flags & ASYNC_TX_FENCE)
dma_flags |= DMA_PREP_FENCE;
/* Since we have clobbered the src_list we are committed /* Drivers force forward progress in case they can not provide
* to doing this asynchronously. Drivers force forward * a descriptor
* progress in case they can not provide a descriptor
*/ */
for (;;) { for (;;) {
dma_dest[0] = unmap->addr[disks - 2];
dma_dest[1] = unmap->addr[disks - 1];
tx = dma->device_prep_dma_pq(chan, dma_dest, tx = dma->device_prep_dma_pq(chan, dma_dest,
&dma_src[src_off], &unmap->addr[src_off],
pq_src_cnt, pq_src_cnt,
&coefs[src_off], len, &scfs[src_off], unmap->len,
dma_flags); dma_flags);
if (likely(tx)) if (likely(tx))
break; break;
...@@ -129,6 +101,7 @@ do_async_gen_syndrome(struct dma_chan *chan, struct page **blocks, ...@@ -129,6 +101,7 @@ do_async_gen_syndrome(struct dma_chan *chan, struct page **blocks,
dma_async_issue_pending(chan); dma_async_issue_pending(chan);
} }
dma_set_unmap(tx, unmap);
async_tx_submit(chan, tx, submit); async_tx_submit(chan, tx, submit);
submit->depend_tx = tx; submit->depend_tx = tx;
...@@ -188,10 +161,6 @@ do_sync_gen_syndrome(struct page **blocks, unsigned int offset, int disks, ...@@ -188,10 +161,6 @@ do_sync_gen_syndrome(struct page **blocks, unsigned int offset, int disks,
* set to NULL those buffers will be replaced with the raid6_zero_page * set to NULL those buffers will be replaced with the raid6_zero_page
* in the synchronous path and omitted in the hardware-asynchronous * in the synchronous path and omitted in the hardware-asynchronous
* path. * path.
*
* 'blocks' note: if submit->scribble is NULL then the contents of
* 'blocks' may be overwritten to perform address conversions
* (dma_map_page() or page_address()).
*/ */
struct dma_async_tx_descriptor * struct dma_async_tx_descriptor *
async_gen_syndrome(struct page **blocks, unsigned int offset, int disks, async_gen_syndrome(struct page **blocks, unsigned int offset, int disks,
...@@ -202,26 +171,69 @@ async_gen_syndrome(struct page **blocks, unsigned int offset, int disks, ...@@ -202,26 +171,69 @@ async_gen_syndrome(struct page **blocks, unsigned int offset, int disks,
&P(blocks, disks), 2, &P(blocks, disks), 2,
blocks, src_cnt, len); blocks, src_cnt, len);
struct dma_device *device = chan ? chan->device : NULL; struct dma_device *device = chan ? chan->device : NULL;
dma_addr_t *dma_src = NULL; struct dmaengine_unmap_data *unmap = NULL;
BUG_ON(disks > 255 || !(P(blocks, disks) || Q(blocks, disks))); BUG_ON(disks > 255 || !(P(blocks, disks) || Q(blocks, disks)));
if (submit->scribble) if (device)
dma_src = submit->scribble; unmap = dmaengine_get_unmap_data(device->dev, disks, GFP_NOIO);
else if (sizeof(dma_addr_t) <= sizeof(struct page *))
dma_src = (dma_addr_t *) blocks;
if (dma_src && device && if (unmap &&
(src_cnt <= dma_maxpq(device, 0) || (src_cnt <= dma_maxpq(device, 0) ||
dma_maxpq(device, DMA_PREP_CONTINUE) > 0) && dma_maxpq(device, DMA_PREP_CONTINUE) > 0) &&
is_dma_pq_aligned(device, offset, 0, len)) { is_dma_pq_aligned(device, offset, 0, len)) {
struct dma_async_tx_descriptor *tx;
enum dma_ctrl_flags dma_flags = 0;
unsigned char coefs[src_cnt];
int i, j;
/* run the p+q asynchronously */ /* run the p+q asynchronously */
pr_debug("%s: (async) disks: %d len: %zu\n", pr_debug("%s: (async) disks: %d len: %zu\n",
__func__, disks, len); __func__, disks, len);
return do_async_gen_syndrome(chan, blocks, raid6_gfexp, offset,
disks, len, dma_src, submit); /* convert source addresses being careful to collapse 'empty'
* sources and update the coefficients accordingly
*/
unmap->len = len;
for (i = 0, j = 0; i < src_cnt; i++) {
if (blocks[i] == NULL)
continue;
unmap->addr[j] = dma_map_page(device->dev, blocks[i], offset,
len, DMA_TO_DEVICE);
coefs[j] = raid6_gfexp[i];
unmap->to_cnt++;
j++;
}
/*
* DMAs use destinations as sources,
* so use BIDIRECTIONAL mapping
*/
unmap->bidi_cnt++;
if (P(blocks, disks))
unmap->addr[j++] = dma_map_page(device->dev, P(blocks, disks),
offset, len, DMA_BIDIRECTIONAL);
else {
unmap->addr[j++] = 0;
dma_flags |= DMA_PREP_PQ_DISABLE_P;
}
unmap->bidi_cnt++;
if (Q(blocks, disks))
unmap->addr[j++] = dma_map_page(device->dev, Q(blocks, disks),
offset, len, DMA_BIDIRECTIONAL);
else {
unmap->addr[j++] = 0;
dma_flags |= DMA_PREP_PQ_DISABLE_Q;
}
tx = do_async_gen_syndrome(chan, coefs, j, unmap, dma_flags, submit);
dmaengine_unmap_put(unmap);
return tx;
} }
dmaengine_unmap_put(unmap);
/* run the pq synchronously */ /* run the pq synchronously */
pr_debug("%s: (sync) disks: %d len: %zu\n", __func__, disks, len); pr_debug("%s: (sync) disks: %d len: %zu\n", __func__, disks, len);
...@@ -277,50 +289,60 @@ async_syndrome_val(struct page **blocks, unsigned int offset, int disks, ...@@ -277,50 +289,60 @@ async_syndrome_val(struct page **blocks, unsigned int offset, int disks,
struct dma_async_tx_descriptor *tx; struct dma_async_tx_descriptor *tx;
unsigned char coefs[disks-2]; unsigned char coefs[disks-2];
enum dma_ctrl_flags dma_flags = submit->cb_fn ? DMA_PREP_INTERRUPT : 0; enum dma_ctrl_flags dma_flags = submit->cb_fn ? DMA_PREP_INTERRUPT : 0;
dma_addr_t *dma_src = NULL; struct dmaengine_unmap_data *unmap = NULL;
int src_cnt = 0;
BUG_ON(disks < 4); BUG_ON(disks < 4);
if (submit->scribble) if (device)
dma_src = submit->scribble; unmap = dmaengine_get_unmap_data(device->dev, disks, GFP_NOIO);
else if (sizeof(dma_addr_t) <= sizeof(struct page *))
dma_src = (dma_addr_t *) blocks;
if (dma_src && device && disks <= dma_maxpq(device, 0) && if (unmap && disks <= dma_maxpq(device, 0) &&
is_dma_pq_aligned(device, offset, 0, len)) { is_dma_pq_aligned(device, offset, 0, len)) {
struct device *dev = device->dev; struct device *dev = device->dev;
dma_addr_t *pq = &dma_src[disks-2]; dma_addr_t pq[2];
int i; int i, j = 0, src_cnt = 0;
pr_debug("%s: (async) disks: %d len: %zu\n", pr_debug("%s: (async) disks: %d len: %zu\n",
__func__, disks, len); __func__, disks, len);
if (!P(blocks, disks))
unmap->len = len;
for (i = 0; i < disks-2; i++)
if (likely(blocks[i])) {
unmap->addr[j] = dma_map_page(dev, blocks[i],
offset, len,
DMA_TO_DEVICE);
coefs[j] = raid6_gfexp[i];
unmap->to_cnt++;
src_cnt++;
j++;
}
if (!P(blocks, disks)) {
pq[0] = 0;
dma_flags |= DMA_PREP_PQ_DISABLE_P; dma_flags |= DMA_PREP_PQ_DISABLE_P;
else } else {
pq[0] = dma_map_page(dev, P(blocks, disks), pq[0] = dma_map_page(dev, P(blocks, disks),
offset, len, offset, len,
DMA_TO_DEVICE); DMA_TO_DEVICE);
if (!Q(blocks, disks)) unmap->addr[j++] = pq[0];
unmap->to_cnt++;
}
if (!Q(blocks, disks)) {
pq[1] = 0;
dma_flags |= DMA_PREP_PQ_DISABLE_Q; dma_flags |= DMA_PREP_PQ_DISABLE_Q;
else } else {
pq[1] = dma_map_page(dev, Q(blocks, disks), pq[1] = dma_map_page(dev, Q(blocks, disks),
offset, len, offset, len,
DMA_TO_DEVICE); DMA_TO_DEVICE);
unmap->addr[j++] = pq[1];
unmap->to_cnt++;
}
if (submit->flags & ASYNC_TX_FENCE) if (submit->flags & ASYNC_TX_FENCE)
dma_flags |= DMA_PREP_FENCE; dma_flags |= DMA_PREP_FENCE;
for (i = 0; i < disks-2; i++)
if (likely(blocks[i])) {
dma_src[src_cnt] = dma_map_page(dev, blocks[i],
offset, len,
DMA_TO_DEVICE);
coefs[src_cnt] = raid6_gfexp[i];
src_cnt++;
}
for (;;) { for (;;) {
tx = device->device_prep_dma_pq_val(chan, pq, dma_src, tx = device->device_prep_dma_pq_val(chan, pq,
unmap->addr,
src_cnt, src_cnt,
coefs, coefs,
len, pqres, len, pqres,
...@@ -330,6 +352,8 @@ async_syndrome_val(struct page **blocks, unsigned int offset, int disks, ...@@ -330,6 +352,8 @@ async_syndrome_val(struct page **blocks, unsigned int offset, int disks,
async_tx_quiesce(&submit->depend_tx); async_tx_quiesce(&submit->depend_tx);
dma_async_issue_pending(chan); dma_async_issue_pending(chan);
} }
dma_set_unmap(tx, unmap);
async_tx_submit(chan, tx, submit); async_tx_submit(chan, tx, submit);
return tx; return tx;
......
...@@ -26,6 +26,7 @@ ...@@ -26,6 +26,7 @@
#include <linux/dma-mapping.h> #include <linux/dma-mapping.h>
#include <linux/raid/pq.h> #include <linux/raid/pq.h>
#include <linux/async_tx.h> #include <linux/async_tx.h>
#include <linux/dmaengine.h>
static struct dma_async_tx_descriptor * static struct dma_async_tx_descriptor *
async_sum_product(struct page *dest, struct page **srcs, unsigned char *coef, async_sum_product(struct page *dest, struct page **srcs, unsigned char *coef,
...@@ -34,35 +35,45 @@ async_sum_product(struct page *dest, struct page **srcs, unsigned char *coef, ...@@ -34,35 +35,45 @@ async_sum_product(struct page *dest, struct page **srcs, unsigned char *coef,
struct dma_chan *chan = async_tx_find_channel(submit, DMA_PQ, struct dma_chan *chan = async_tx_find_channel(submit, DMA_PQ,
&dest, 1, srcs, 2, len); &dest, 1, srcs, 2, len);
struct dma_device *dma = chan ? chan->device : NULL; struct dma_device *dma = chan ? chan->device : NULL;
struct dmaengine_unmap_data *unmap = NULL;
const u8 *amul, *bmul; const u8 *amul, *bmul;
u8 ax, bx; u8 ax, bx;
u8 *a, *b, *c; u8 *a, *b, *c;
if (dma) { if (dma)
dma_addr_t dma_dest[2]; unmap = dmaengine_get_unmap_data(dma->dev, 3, GFP_NOIO);
dma_addr_t dma_src[2];
if (unmap) {
struct device *dev = dma->dev; struct device *dev = dma->dev;
dma_addr_t pq[2];
struct dma_async_tx_descriptor *tx; struct dma_async_tx_descriptor *tx;
enum dma_ctrl_flags dma_flags = DMA_PREP_PQ_DISABLE_P; enum dma_ctrl_flags dma_flags = DMA_PREP_PQ_DISABLE_P;
if (submit->flags & ASYNC_TX_FENCE) if (submit->flags & ASYNC_TX_FENCE)
dma_flags |= DMA_PREP_FENCE; dma_flags |= DMA_PREP_FENCE;
dma_dest[1] = dma_map_page(dev, dest, 0, len, DMA_BIDIRECTIONAL); unmap->addr[0] = dma_map_page(dev, srcs[0], 0, len, DMA_TO_DEVICE);
dma_src[0] = dma_map_page(dev, srcs[0], 0, len, DMA_TO_DEVICE); unmap->addr[1] = dma_map_page(dev, srcs[1], 0, len, DMA_TO_DEVICE);
dma_src[1] = dma_map_page(dev, srcs[1], 0, len, DMA_TO_DEVICE); unmap->to_cnt = 2;
tx = dma->device_prep_dma_pq(chan, dma_dest, dma_src, 2, coef,
unmap->addr[2] = dma_map_page(dev, dest, 0, len, DMA_BIDIRECTIONAL);
unmap->bidi_cnt = 1;
/* engine only looks at Q, but expects it to follow P */
pq[1] = unmap->addr[2];
unmap->len = len;
tx = dma->device_prep_dma_pq(chan, pq, unmap->addr, 2, coef,
len, dma_flags); len, dma_flags);
if (tx) { if (tx) {
dma_set_unmap(tx, unmap);
async_tx_submit(chan, tx, submit); async_tx_submit(chan, tx, submit);
dmaengine_unmap_put(unmap);
return tx; return tx;
} }
/* could not get a descriptor, unmap and fall through to /* could not get a descriptor, unmap and fall through to
* the synchronous path * the synchronous path
*/ */
dma_unmap_page(dev, dma_dest[1], len, DMA_BIDIRECTIONAL); dmaengine_unmap_put(unmap);
dma_unmap_page(dev, dma_src[0], len, DMA_TO_DEVICE);
dma_unmap_page(dev, dma_src[1], len, DMA_TO_DEVICE);
} }
/* run the operation synchronously */ /* run the operation synchronously */
...@@ -89,23 +100,38 @@ async_mult(struct page *dest, struct page *src, u8 coef, size_t len, ...@@ -89,23 +100,38 @@ async_mult(struct page *dest, struct page *src, u8 coef, size_t len,
struct dma_chan *chan = async_tx_find_channel(submit, DMA_PQ, struct dma_chan *chan = async_tx_find_channel(submit, DMA_PQ,
&dest, 1, &src, 1, len); &dest, 1, &src, 1, len);
struct dma_device *dma = chan ? chan->device : NULL; struct dma_device *dma = chan ? chan->device : NULL;
struct dmaengine_unmap_data *unmap = NULL;
const u8 *qmul; /* Q multiplier table */ const u8 *qmul; /* Q multiplier table */
u8 *d, *s; u8 *d, *s;
if (dma) { if (dma)
unmap = dmaengine_get_unmap_data(dma->dev, 3, GFP_NOIO);
if (unmap) {
dma_addr_t dma_dest[2]; dma_addr_t dma_dest[2];
dma_addr_t dma_src[1];
struct device *dev = dma->dev; struct device *dev = dma->dev;
struct dma_async_tx_descriptor *tx; struct dma_async_tx_descriptor *tx;
enum dma_ctrl_flags dma_flags = DMA_PREP_PQ_DISABLE_P; enum dma_ctrl_flags dma_flags = DMA_PREP_PQ_DISABLE_P;
if (submit->flags & ASYNC_TX_FENCE) if (submit->flags & ASYNC_TX_FENCE)
dma_flags |= DMA_PREP_FENCE; dma_flags |= DMA_PREP_FENCE;
dma_dest[1] = dma_map_page(dev, dest, 0, len, DMA_BIDIRECTIONAL); unmap->addr[0] = dma_map_page(dev, src, 0, len, DMA_TO_DEVICE);
dma_src[0] = dma_map_page(dev, src, 0, len, DMA_TO_DEVICE); unmap->to_cnt++;
tx = dma->device_prep_dma_pq(chan, dma_dest, dma_src, 1, &coef, unmap->addr[1] = dma_map_page(dev, dest, 0, len, DMA_BIDIRECTIONAL);
len, dma_flags); dma_dest[1] = unmap->addr[1];
unmap->bidi_cnt++;
unmap->len = len;
/* this looks funny, but the engine looks for Q at
* dma_dest[1] and ignores dma_dest[0] as a dest
* due to DMA_PREP_PQ_DISABLE_P
*/
tx = dma->device_prep_dma_pq(chan, dma_dest, unmap->addr,
1, &coef, len, dma_flags);
if (tx) { if (tx) {
dma_set_unmap(tx, unmap);
dmaengine_unmap_put(unmap);
async_tx_submit(chan, tx, submit); async_tx_submit(chan, tx, submit);
return tx; return tx;
} }
...@@ -113,8 +139,7 @@ async_mult(struct page *dest, struct page *src, u8 coef, size_t len, ...@@ -113,8 +139,7 @@ async_mult(struct page *dest, struct page *src, u8 coef, size_t len,
/* could not get a descriptor, unmap and fall through to /* could not get a descriptor, unmap and fall through to
* the synchronous path * the synchronous path
*/ */
dma_unmap_page(dev, dma_dest[1], len, DMA_BIDIRECTIONAL); dmaengine_unmap_put(unmap);
dma_unmap_page(dev, dma_src[0], len, DMA_TO_DEVICE);
} }
/* no channel available, or failed to allocate a descriptor, so /* no channel available, or failed to allocate a descriptor, so
......
...@@ -128,7 +128,7 @@ async_tx_channel_switch(struct dma_async_tx_descriptor *depend_tx, ...@@ -128,7 +128,7 @@ async_tx_channel_switch(struct dma_async_tx_descriptor *depend_tx,
} }
device->device_issue_pending(chan); device->device_issue_pending(chan);
} else { } else {
if (dma_wait_for_async_tx(depend_tx) != DMA_SUCCESS) if (dma_wait_for_async_tx(depend_tx) != DMA_COMPLETE)
panic("%s: DMA error waiting for depend_tx\n", panic("%s: DMA error waiting for depend_tx\n",
__func__); __func__);
tx->tx_submit(tx); tx->tx_submit(tx);
...@@ -280,7 +280,7 @@ void async_tx_quiesce(struct dma_async_tx_descriptor **tx) ...@@ -280,7 +280,7 @@ void async_tx_quiesce(struct dma_async_tx_descriptor **tx)
* we are referring to the correct operation * we are referring to the correct operation
*/ */
BUG_ON(async_tx_test_ack(*tx)); BUG_ON(async_tx_test_ack(*tx));
if (dma_wait_for_async_tx(*tx) != DMA_SUCCESS) if (dma_wait_for_async_tx(*tx) != DMA_COMPLETE)
panic("%s: DMA error waiting for transaction\n", panic("%s: DMA error waiting for transaction\n",
__func__); __func__);
async_tx_ack(*tx); async_tx_ack(*tx);
......
...@@ -33,48 +33,31 @@ ...@@ -33,48 +33,31 @@
/* do_async_xor - dma map the pages and perform the xor with an engine */ /* do_async_xor - dma map the pages and perform the xor with an engine */
static __async_inline struct dma_async_tx_descriptor * static __async_inline struct dma_async_tx_descriptor *
do_async_xor(struct dma_chan *chan, struct page *dest, struct page **src_list, do_async_xor(struct dma_chan *chan, struct dmaengine_unmap_data *unmap,
unsigned int offset, int src_cnt, size_t len, dma_addr_t *dma_src,
struct async_submit_ctl *submit) struct async_submit_ctl *submit)
{ {
struct dma_device *dma = chan->device; struct dma_device *dma = chan->device;
struct dma_async_tx_descriptor *tx = NULL; struct dma_async_tx_descriptor *tx = NULL;
int src_off = 0;
int i;
dma_async_tx_callback cb_fn_orig = submit->cb_fn; dma_async_tx_callback cb_fn_orig = submit->cb_fn;
void *cb_param_orig = submit->cb_param; void *cb_param_orig = submit->cb_param;
enum async_tx_flags flags_orig = submit->flags; enum async_tx_flags flags_orig = submit->flags;
enum dma_ctrl_flags dma_flags; enum dma_ctrl_flags dma_flags = 0;
int xor_src_cnt = 0; int src_cnt = unmap->to_cnt;
dma_addr_t dma_dest; int xor_src_cnt;
dma_addr_t dma_dest = unmap->addr[unmap->to_cnt];
/* map the dest bidrectional in case it is re-used as a source */ dma_addr_t *src_list = unmap->addr;
dma_dest = dma_map_page(dma->dev, dest, offset, len, DMA_BIDIRECTIONAL);
for (i = 0; i < src_cnt; i++) {
/* only map the dest once */
if (!src_list[i])
continue;
if (unlikely(src_list[i] == dest)) {
dma_src[xor_src_cnt++] = dma_dest;
continue;
}
dma_src[xor_src_cnt++] = dma_map_page(dma->dev, src_list[i], offset,
len, DMA_TO_DEVICE);
}
src_cnt = xor_src_cnt;
while (src_cnt) { while (src_cnt) {
dma_addr_t tmp;
submit->flags = flags_orig; submit->flags = flags_orig;
dma_flags = 0;
xor_src_cnt = min(src_cnt, (int)dma->max_xor); xor_src_cnt = min(src_cnt, (int)dma->max_xor);
/* if we are submitting additional xors, leave the chain open, /* if we are submitting additional xors, leave the chain open
* clear the callback parameters, and leave the destination * and clear the callback parameters
* buffer mapped
*/ */
if (src_cnt > xor_src_cnt) { if (src_cnt > xor_src_cnt) {
submit->flags &= ~ASYNC_TX_ACK; submit->flags &= ~ASYNC_TX_ACK;
submit->flags |= ASYNC_TX_FENCE; submit->flags |= ASYNC_TX_FENCE;
dma_flags = DMA_COMPL_SKIP_DEST_UNMAP;
submit->cb_fn = NULL; submit->cb_fn = NULL;
submit->cb_param = NULL; submit->cb_param = NULL;
} else { } else {
...@@ -85,12 +68,18 @@ do_async_xor(struct dma_chan *chan, struct page *dest, struct page **src_list, ...@@ -85,12 +68,18 @@ do_async_xor(struct dma_chan *chan, struct page *dest, struct page **src_list,
dma_flags |= DMA_PREP_INTERRUPT; dma_flags |= DMA_PREP_INTERRUPT;
if (submit->flags & ASYNC_TX_FENCE) if (submit->flags & ASYNC_TX_FENCE)
dma_flags |= DMA_PREP_FENCE; dma_flags |= DMA_PREP_FENCE;
/* Since we have clobbered the src_list we are committed
* to doing this asynchronously. Drivers force forward progress /* Drivers force forward progress in case they can not provide a
* in case they can not provide a descriptor * descriptor
*/ */
tx = dma->device_prep_dma_xor(chan, dma_dest, &dma_src[src_off], tmp = src_list[0];
xor_src_cnt, len, dma_flags); if (src_list > unmap->addr)
src_list[0] = dma_dest;
tx = dma->device_prep_dma_xor(chan, dma_dest, src_list,
xor_src_cnt, unmap->len,
dma_flags);
src_list[0] = tmp;
if (unlikely(!tx)) if (unlikely(!tx))
async_tx_quiesce(&submit->depend_tx); async_tx_quiesce(&submit->depend_tx);
...@@ -99,22 +88,21 @@ do_async_xor(struct dma_chan *chan, struct page *dest, struct page **src_list, ...@@ -99,22 +88,21 @@ do_async_xor(struct dma_chan *chan, struct page *dest, struct page **src_list,
while (unlikely(!tx)) { while (unlikely(!tx)) {
dma_async_issue_pending(chan); dma_async_issue_pending(chan);
tx = dma->device_prep_dma_xor(chan, dma_dest, tx = dma->device_prep_dma_xor(chan, dma_dest,
&dma_src[src_off], src_list,
xor_src_cnt, len, xor_src_cnt, unmap->len,
dma_flags); dma_flags);
} }
dma_set_unmap(tx, unmap);
async_tx_submit(chan, tx, submit); async_tx_submit(chan, tx, submit);
submit->depend_tx = tx; submit->depend_tx = tx;
if (src_cnt > xor_src_cnt) { if (src_cnt > xor_src_cnt) {
/* drop completed sources */ /* drop completed sources */
src_cnt -= xor_src_cnt; src_cnt -= xor_src_cnt;
src_off += xor_src_cnt;
/* use the intermediate result a source */ /* use the intermediate result a source */
dma_src[--src_off] = dma_dest;
src_cnt++; src_cnt++;
src_list += xor_src_cnt - 1;
} else } else
break; break;
} }
...@@ -189,22 +177,40 @@ async_xor(struct page *dest, struct page **src_list, unsigned int offset, ...@@ -189,22 +177,40 @@ async_xor(struct page *dest, struct page **src_list, unsigned int offset,
struct dma_chan *chan = async_tx_find_channel(submit, DMA_XOR, struct dma_chan *chan = async_tx_find_channel(submit, DMA_XOR,
&dest, 1, src_list, &dest, 1, src_list,
src_cnt, len); src_cnt, len);
dma_addr_t *dma_src = NULL; struct dma_device *device = chan ? chan->device : NULL;
struct dmaengine_unmap_data *unmap = NULL;
BUG_ON(src_cnt <= 1); BUG_ON(src_cnt <= 1);
if (submit->scribble) if (device)
dma_src = submit->scribble; unmap = dmaengine_get_unmap_data(device->dev, src_cnt+1, GFP_NOIO);
else if (sizeof(dma_addr_t) <= sizeof(struct page *))
dma_src = (dma_addr_t *) src_list; if (unmap && is_dma_xor_aligned(device, offset, 0, len)) {
struct dma_async_tx_descriptor *tx;
int i, j;
if (dma_src && chan && is_dma_xor_aligned(chan->device, offset, 0, len)) {
/* run the xor asynchronously */ /* run the xor asynchronously */
pr_debug("%s (async): len: %zu\n", __func__, len); pr_debug("%s (async): len: %zu\n", __func__, len);
return do_async_xor(chan, dest, src_list, offset, src_cnt, len, unmap->len = len;
dma_src, submit); for (i = 0, j = 0; i < src_cnt; i++) {
if (!src_list[i])
continue;
unmap->to_cnt++;
unmap->addr[j++] = dma_map_page(device->dev, src_list[i],
offset, len, DMA_TO_DEVICE);
}
/* map it bidirectional as it may be re-used as a source */
unmap->addr[j] = dma_map_page(device->dev, dest, offset, len,
DMA_BIDIRECTIONAL);
unmap->bidi_cnt = 1;
tx = do_async_xor(chan, unmap, submit);
dmaengine_unmap_put(unmap);
return tx;
} else { } else {
dmaengine_unmap_put(unmap);
/* run the xor synchronously */ /* run the xor synchronously */
pr_debug("%s (sync): len: %zu\n", __func__, len); pr_debug("%s (sync): len: %zu\n", __func__, len);
WARN_ONCE(chan, "%s: no space for dma address conversion\n", WARN_ONCE(chan, "%s: no space for dma address conversion\n",
...@@ -268,16 +274,14 @@ async_xor_val(struct page *dest, struct page **src_list, unsigned int offset, ...@@ -268,16 +274,14 @@ async_xor_val(struct page *dest, struct page **src_list, unsigned int offset,
struct dma_chan *chan = xor_val_chan(submit, dest, src_list, src_cnt, len); struct dma_chan *chan = xor_val_chan(submit, dest, src_list, src_cnt, len);
struct dma_device *device = chan ? chan->device : NULL; struct dma_device *device = chan ? chan->device : NULL;
struct dma_async_tx_descriptor *tx = NULL; struct dma_async_tx_descriptor *tx = NULL;
dma_addr_t *dma_src = NULL; struct dmaengine_unmap_data *unmap = NULL;
BUG_ON(src_cnt <= 1); BUG_ON(src_cnt <= 1);
if (submit->scribble) if (device)
dma_src = submit->scribble; unmap = dmaengine_get_unmap_data(device->dev, src_cnt, GFP_NOIO);
else if (sizeof(dma_addr_t) <= sizeof(struct page *))
dma_src = (dma_addr_t *) src_list;
if (dma_src && device && src_cnt <= device->max_xor && if (unmap && src_cnt <= device->max_xor &&
is_dma_xor_aligned(device, offset, 0, len)) { is_dma_xor_aligned(device, offset, 0, len)) {
unsigned long dma_prep_flags = 0; unsigned long dma_prep_flags = 0;
int i; int i;
...@@ -288,11 +292,15 @@ async_xor_val(struct page *dest, struct page **src_list, unsigned int offset, ...@@ -288,11 +292,15 @@ async_xor_val(struct page *dest, struct page **src_list, unsigned int offset,
dma_prep_flags |= DMA_PREP_INTERRUPT; dma_prep_flags |= DMA_PREP_INTERRUPT;
if (submit->flags & ASYNC_TX_FENCE) if (submit->flags & ASYNC_TX_FENCE)
dma_prep_flags |= DMA_PREP_FENCE; dma_prep_flags |= DMA_PREP_FENCE;
for (i = 0; i < src_cnt; i++)
dma_src[i] = dma_map_page(device->dev, src_list[i],
offset, len, DMA_TO_DEVICE);
tx = device->device_prep_dma_xor_val(chan, dma_src, src_cnt, for (i = 0; i < src_cnt; i++) {
unmap->addr[i] = dma_map_page(device->dev, src_list[i],
offset, len, DMA_TO_DEVICE);
unmap->to_cnt++;
}
unmap->len = len;
tx = device->device_prep_dma_xor_val(chan, unmap->addr, src_cnt,
len, result, len, result,
dma_prep_flags); dma_prep_flags);
if (unlikely(!tx)) { if (unlikely(!tx)) {
...@@ -301,11 +309,11 @@ async_xor_val(struct page *dest, struct page **src_list, unsigned int offset, ...@@ -301,11 +309,11 @@ async_xor_val(struct page *dest, struct page **src_list, unsigned int offset,
while (!tx) { while (!tx) {
dma_async_issue_pending(chan); dma_async_issue_pending(chan);
tx = device->device_prep_dma_xor_val(chan, tx = device->device_prep_dma_xor_val(chan,
dma_src, src_cnt, len, result, unmap->addr, src_cnt, len, result,
dma_prep_flags); dma_prep_flags);
} }
} }
dma_set_unmap(tx, unmap);
async_tx_submit(chan, tx, submit); async_tx_submit(chan, tx, submit);
} else { } else {
enum async_tx_flags flags_orig = submit->flags; enum async_tx_flags flags_orig = submit->flags;
...@@ -327,6 +335,7 @@ async_xor_val(struct page *dest, struct page **src_list, unsigned int offset, ...@@ -327,6 +335,7 @@ async_xor_val(struct page *dest, struct page **src_list, unsigned int offset,
async_tx_sync_epilog(submit); async_tx_sync_epilog(submit);
submit->flags = flags_orig; submit->flags = flags_orig;
} }
dmaengine_unmap_put(unmap);
return tx; return tx;
} }
......
...@@ -28,7 +28,7 @@ ...@@ -28,7 +28,7 @@
#undef pr #undef pr
#define pr(fmt, args...) pr_info("raid6test: " fmt, ##args) #define pr(fmt, args...) pr_info("raid6test: " fmt, ##args)
#define NDISKS 16 /* Including P and Q */ #define NDISKS 64 /* Including P and Q */
static struct page *dataptrs[NDISKS]; static struct page *dataptrs[NDISKS];
static addr_conv_t addr_conv[NDISKS]; static addr_conv_t addr_conv[NDISKS];
...@@ -219,6 +219,14 @@ static int raid6_test(void) ...@@ -219,6 +219,14 @@ static int raid6_test(void)
err += test(11, &tests); err += test(11, &tests);
err += test(12, &tests); err += test(12, &tests);
} }
/* the 24 disk case is special for ioatdma as it is the boudary point
* at which it needs to switch from 8-source ops to 16-source
* ops for continuation (assumes DMA_HAS_PQ_CONTINUE is not set)
*/
if (NDISKS > 24)
err += test(24, &tests);
err += test(NDISKS, &tests); err += test(NDISKS, &tests);
pr("\n"); pr("\n");
......
...@@ -396,8 +396,7 @@ dma_xfer(struct arasan_cf_dev *acdev, dma_addr_t src, dma_addr_t dest, u32 len) ...@@ -396,8 +396,7 @@ dma_xfer(struct arasan_cf_dev *acdev, dma_addr_t src, dma_addr_t dest, u32 len)
struct dma_async_tx_descriptor *tx; struct dma_async_tx_descriptor *tx;
struct dma_chan *chan = acdev->dma_chan; struct dma_chan *chan = acdev->dma_chan;
dma_cookie_t cookie; dma_cookie_t cookie;
unsigned long flags = DMA_PREP_INTERRUPT | DMA_COMPL_SKIP_SRC_UNMAP | unsigned long flags = DMA_PREP_INTERRUPT;
DMA_COMPL_SKIP_DEST_UNMAP;
int ret = 0; int ret = 0;
tx = chan->device->device_prep_dma_memcpy(chan, dest, src, len, flags); tx = chan->device->device_prep_dma_memcpy(chan, dest, src, len, flags);
......
...@@ -89,14 +89,15 @@ config AT_HDMAC ...@@ -89,14 +89,15 @@ config AT_HDMAC
Support the Atmel AHB DMA controller. Support the Atmel AHB DMA controller.
config FSL_DMA config FSL_DMA
tristate "Freescale Elo and Elo Plus DMA support" tristate "Freescale Elo series DMA support"
depends on FSL_SOC depends on FSL_SOC
select DMA_ENGINE select DMA_ENGINE
select ASYNC_TX_ENABLE_CHANNEL_SWITCH select ASYNC_TX_ENABLE_CHANNEL_SWITCH
---help--- ---help---
Enable support for the Freescale Elo and Elo Plus DMA controllers. Enable support for the Freescale Elo series DMA controllers.
The Elo is the DMA controller on some 82xx and 83xx parts, and the The Elo is the DMA controller on some mpc82xx and mpc83xx parts, the
Elo Plus is the DMA controller on 85xx and 86xx parts. EloPlus is on mpc85xx and mpc86xx and Pxxx parts, and the Elo3 is on
some Txxx and Bxxx parts.
config MPC512X_DMA config MPC512X_DMA
tristate "Freescale MPC512x built-in DMA engine support" tristate "Freescale MPC512x built-in DMA engine support"
......
...@@ -1164,42 +1164,12 @@ static void pl08x_free_txd(struct pl08x_driver_data *pl08x, ...@@ -1164,42 +1164,12 @@ static void pl08x_free_txd(struct pl08x_driver_data *pl08x,
kfree(txd); kfree(txd);
} }
static void pl08x_unmap_buffers(struct pl08x_txd *txd)
{
struct device *dev = txd->vd.tx.chan->device->dev;
struct pl08x_sg *dsg;
if (!(txd->vd.tx.flags & DMA_COMPL_SKIP_SRC_UNMAP)) {
if (txd->vd.tx.flags & DMA_COMPL_SRC_UNMAP_SINGLE)
list_for_each_entry(dsg, &txd->dsg_list, node)
dma_unmap_single(dev, dsg->src_addr, dsg->len,
DMA_TO_DEVICE);
else {
list_for_each_entry(dsg, &txd->dsg_list, node)
dma_unmap_page(dev, dsg->src_addr, dsg->len,
DMA_TO_DEVICE);
}
}
if (!(txd->vd.tx.flags & DMA_COMPL_SKIP_DEST_UNMAP)) {
if (txd->vd.tx.flags & DMA_COMPL_DEST_UNMAP_SINGLE)
list_for_each_entry(dsg, &txd->dsg_list, node)
dma_unmap_single(dev, dsg->dst_addr, dsg->len,
DMA_FROM_DEVICE);
else
list_for_each_entry(dsg, &txd->dsg_list, node)
dma_unmap_page(dev, dsg->dst_addr, dsg->len,
DMA_FROM_DEVICE);
}
}
static void pl08x_desc_free(struct virt_dma_desc *vd) static void pl08x_desc_free(struct virt_dma_desc *vd)
{ {
struct pl08x_txd *txd = to_pl08x_txd(&vd->tx); struct pl08x_txd *txd = to_pl08x_txd(&vd->tx);
struct pl08x_dma_chan *plchan = to_pl08x_chan(vd->tx.chan); struct pl08x_dma_chan *plchan = to_pl08x_chan(vd->tx.chan);
if (!plchan->slave) dma_descriptor_unmap(txd);
pl08x_unmap_buffers(txd);
if (!txd->done) if (!txd->done)
pl08x_release_mux(plchan); pl08x_release_mux(plchan);
...@@ -1252,7 +1222,7 @@ static enum dma_status pl08x_dma_tx_status(struct dma_chan *chan, ...@@ -1252,7 +1222,7 @@ static enum dma_status pl08x_dma_tx_status(struct dma_chan *chan,
size_t bytes = 0; size_t bytes = 0;
ret = dma_cookie_status(chan, cookie, txstate); ret = dma_cookie_status(chan, cookie, txstate);
if (ret == DMA_SUCCESS) if (ret == DMA_COMPLETE)
return ret; return ret;
/* /*
...@@ -1267,7 +1237,7 @@ static enum dma_status pl08x_dma_tx_status(struct dma_chan *chan, ...@@ -1267,7 +1237,7 @@ static enum dma_status pl08x_dma_tx_status(struct dma_chan *chan,
spin_lock_irqsave(&plchan->vc.lock, flags); spin_lock_irqsave(&plchan->vc.lock, flags);
ret = dma_cookie_status(chan, cookie, txstate); ret = dma_cookie_status(chan, cookie, txstate);
if (ret != DMA_SUCCESS) { if (ret != DMA_COMPLETE) {
vd = vchan_find_desc(&plchan->vc, cookie); vd = vchan_find_desc(&plchan->vc, cookie);
if (vd) { if (vd) {
/* On the issued list, so hasn't been processed yet */ /* On the issued list, so hasn't been processed yet */
...@@ -2138,8 +2108,7 @@ static int pl08x_probe(struct amba_device *adev, const struct amba_id *id) ...@@ -2138,8 +2108,7 @@ static int pl08x_probe(struct amba_device *adev, const struct amba_id *id)
writel(0x000000FF, pl08x->base + PL080_ERR_CLEAR); writel(0x000000FF, pl08x->base + PL080_ERR_CLEAR);
writel(0x000000FF, pl08x->base + PL080_TC_CLEAR); writel(0x000000FF, pl08x->base + PL080_TC_CLEAR);
ret = request_irq(adev->irq[0], pl08x_irq, IRQF_DISABLED, ret = request_irq(adev->irq[0], pl08x_irq, 0, DRIVER_NAME, pl08x);
DRIVER_NAME, pl08x);
if (ret) { if (ret) {
dev_err(&adev->dev, "%s failed to request interrupt %d\n", dev_err(&adev->dev, "%s failed to request interrupt %d\n",
__func__, adev->irq[0]); __func__, adev->irq[0]);
......
...@@ -344,31 +344,7 @@ atc_chain_complete(struct at_dma_chan *atchan, struct at_desc *desc) ...@@ -344,31 +344,7 @@ atc_chain_complete(struct at_dma_chan *atchan, struct at_desc *desc)
/* move myself to free_list */ /* move myself to free_list */
list_move(&desc->desc_node, &atchan->free_list); list_move(&desc->desc_node, &atchan->free_list);
/* unmap dma addresses (not on slave channels) */ dma_descriptor_unmap(txd);
if (!atchan->chan_common.private) {
struct device *parent = chan2parent(&atchan->chan_common);
if (!(txd->flags & DMA_COMPL_SKIP_DEST_UNMAP)) {
if (txd->flags & DMA_COMPL_DEST_UNMAP_SINGLE)
dma_unmap_single(parent,
desc->lli.daddr,
desc->len, DMA_FROM_DEVICE);
else
dma_unmap_page(parent,
desc->lli.daddr,
desc->len, DMA_FROM_DEVICE);
}
if (!(txd->flags & DMA_COMPL_SKIP_SRC_UNMAP)) {
if (txd->flags & DMA_COMPL_SRC_UNMAP_SINGLE)
dma_unmap_single(parent,
desc->lli.saddr,
desc->len, DMA_TO_DEVICE);
else
dma_unmap_page(parent,
desc->lli.saddr,
desc->len, DMA_TO_DEVICE);
}
}
/* for cyclic transfers, /* for cyclic transfers,
* no need to replay callback function while stopping */ * no need to replay callback function while stopping */
if (!atc_chan_is_cyclic(atchan)) { if (!atc_chan_is_cyclic(atchan)) {
...@@ -1102,7 +1078,7 @@ atc_tx_status(struct dma_chan *chan, ...@@ -1102,7 +1078,7 @@ atc_tx_status(struct dma_chan *chan,
int bytes = 0; int bytes = 0;
ret = dma_cookie_status(chan, cookie, txstate); ret = dma_cookie_status(chan, cookie, txstate);
if (ret == DMA_SUCCESS) if (ret == DMA_COMPLETE)
return ret; return ret;
/* /*
* There's no point calculating the residue if there's * There's no point calculating the residue if there's
......
...@@ -2369,7 +2369,7 @@ coh901318_tx_status(struct dma_chan *chan, dma_cookie_t cookie, ...@@ -2369,7 +2369,7 @@ coh901318_tx_status(struct dma_chan *chan, dma_cookie_t cookie,
enum dma_status ret; enum dma_status ret;
ret = dma_cookie_status(chan, cookie, txstate); ret = dma_cookie_status(chan, cookie, txstate);
if (ret == DMA_SUCCESS) if (ret == DMA_COMPLETE)
return ret; return ret;
dma_set_residue(txstate, coh901318_get_bytes_left(chan)); dma_set_residue(txstate, coh901318_get_bytes_left(chan));
...@@ -2694,7 +2694,7 @@ static int __init coh901318_probe(struct platform_device *pdev) ...@@ -2694,7 +2694,7 @@ static int __init coh901318_probe(struct platform_device *pdev)
if (irq < 0) if (irq < 0)
return irq; return irq;
err = devm_request_irq(&pdev->dev, irq, dma_irq_handler, IRQF_DISABLED, err = devm_request_irq(&pdev->dev, irq, dma_irq_handler, 0,
"coh901318", base); "coh901318", base);
if (err) if (err)
return err; return err;
......
This diff is collapsed.
...@@ -491,7 +491,7 @@ static enum dma_status jz4740_dma_tx_status(struct dma_chan *c, ...@@ -491,7 +491,7 @@ static enum dma_status jz4740_dma_tx_status(struct dma_chan *c,
unsigned long flags; unsigned long flags;
status = dma_cookie_status(c, cookie, state); status = dma_cookie_status(c, cookie, state);
if (status == DMA_SUCCESS || !state) if (status == DMA_COMPLETE || !state)
return status; return status;
spin_lock_irqsave(&chan->vchan.lock, flags); spin_lock_irqsave(&chan->vchan.lock, flags);
......
...@@ -65,6 +65,7 @@ ...@@ -65,6 +65,7 @@
#include <linux/acpi.h> #include <linux/acpi.h>
#include <linux/acpi_dma.h> #include <linux/acpi_dma.h>
#include <linux/of_dma.h> #include <linux/of_dma.h>
#include <linux/mempool.h>
static DEFINE_MUTEX(dma_list_mutex); static DEFINE_MUTEX(dma_list_mutex);
static DEFINE_IDR(dma_idr); static DEFINE_IDR(dma_idr);
...@@ -901,98 +902,132 @@ void dma_async_device_unregister(struct dma_device *device) ...@@ -901,98 +902,132 @@ void dma_async_device_unregister(struct dma_device *device)
} }
EXPORT_SYMBOL(dma_async_device_unregister); EXPORT_SYMBOL(dma_async_device_unregister);
/** struct dmaengine_unmap_pool {
* dma_async_memcpy_buf_to_buf - offloaded copy between virtual addresses struct kmem_cache *cache;
* @chan: DMA channel to offload copy to const char *name;
* @dest: destination address (virtual) mempool_t *pool;
* @src: source address (virtual) size_t size;
* @len: length };
*
* Both @dest and @src must be mappable to a bus address according to the
* DMA mapping API rules for streaming mappings.
* Both @dest and @src must stay memory resident (kernel memory or locked
* user space pages).
*/
dma_cookie_t
dma_async_memcpy_buf_to_buf(struct dma_chan *chan, void *dest,
void *src, size_t len)
{
struct dma_device *dev = chan->device;
struct dma_async_tx_descriptor *tx;
dma_addr_t dma_dest, dma_src;
dma_cookie_t cookie;
unsigned long flags;
dma_src = dma_map_single(dev->dev, src, len, DMA_TO_DEVICE); #define __UNMAP_POOL(x) { .size = x, .name = "dmaengine-unmap-" __stringify(x) }
dma_dest = dma_map_single(dev->dev, dest, len, DMA_FROM_DEVICE); static struct dmaengine_unmap_pool unmap_pool[] = {
flags = DMA_CTRL_ACK | __UNMAP_POOL(2),
DMA_COMPL_SRC_UNMAP_SINGLE | #if IS_ENABLED(CONFIG_ASYNC_TX_DMA)
DMA_COMPL_DEST_UNMAP_SINGLE; __UNMAP_POOL(16),
tx = dev->device_prep_dma_memcpy(chan, dma_dest, dma_src, len, flags); __UNMAP_POOL(128),
__UNMAP_POOL(256),
#endif
};
if (!tx) { static struct dmaengine_unmap_pool *__get_unmap_pool(int nr)
dma_unmap_single(dev->dev, dma_src, len, DMA_TO_DEVICE); {
dma_unmap_single(dev->dev, dma_dest, len, DMA_FROM_DEVICE); int order = get_count_order(nr);
return -ENOMEM;
switch (order) {
case 0 ... 1:
return &unmap_pool[0];
case 2 ... 4:
return &unmap_pool[1];
case 5 ... 7:
return &unmap_pool[2];
case 8:
return &unmap_pool[3];
default:
BUG();
return NULL;
} }
}
tx->callback = NULL; static void dmaengine_unmap(struct kref *kref)
cookie = tx->tx_submit(tx); {
struct dmaengine_unmap_data *unmap = container_of(kref, typeof(*unmap), kref);
struct device *dev = unmap->dev;
int cnt, i;
cnt = unmap->to_cnt;
for (i = 0; i < cnt; i++)
dma_unmap_page(dev, unmap->addr[i], unmap->len,
DMA_TO_DEVICE);
cnt += unmap->from_cnt;
for (; i < cnt; i++)
dma_unmap_page(dev, unmap->addr[i], unmap->len,
DMA_FROM_DEVICE);
cnt += unmap->bidi_cnt;
for (; i < cnt; i++) {
if (unmap->addr[i] == 0)
continue;
dma_unmap_page(dev, unmap->addr[i], unmap->len,
DMA_BIDIRECTIONAL);
}
mempool_free(unmap, __get_unmap_pool(cnt)->pool);
}
preempt_disable(); void dmaengine_unmap_put(struct dmaengine_unmap_data *unmap)
__this_cpu_add(chan->local->bytes_transferred, len); {
__this_cpu_inc(chan->local->memcpy_count); if (unmap)
preempt_enable(); kref_put(&unmap->kref, dmaengine_unmap);
}
EXPORT_SYMBOL_GPL(dmaengine_unmap_put);
return cookie; static void dmaengine_destroy_unmap_pool(void)
{
int i;
for (i = 0; i < ARRAY_SIZE(unmap_pool); i++) {
struct dmaengine_unmap_pool *p = &unmap_pool[i];
if (p->pool)
mempool_destroy(p->pool);
p->pool = NULL;
if (p->cache)
kmem_cache_destroy(p->cache);
p->cache = NULL;
}
} }
EXPORT_SYMBOL(dma_async_memcpy_buf_to_buf);
/** static int __init dmaengine_init_unmap_pool(void)
* dma_async_memcpy_buf_to_pg - offloaded copy from address to page
* @chan: DMA channel to offload copy to
* @page: destination page
* @offset: offset in page to copy to
* @kdata: source address (virtual)
* @len: length
*
* Both @page/@offset and @kdata must be mappable to a bus address according
* to the DMA mapping API rules for streaming mappings.
* Both @page/@offset and @kdata must stay memory resident (kernel memory or
* locked user space pages)
*/
dma_cookie_t
dma_async_memcpy_buf_to_pg(struct dma_chan *chan, struct page *page,
unsigned int offset, void *kdata, size_t len)
{ {
struct dma_device *dev = chan->device; int i;
struct dma_async_tx_descriptor *tx;
dma_addr_t dma_dest, dma_src;
dma_cookie_t cookie;
unsigned long flags;
dma_src = dma_map_single(dev->dev, kdata, len, DMA_TO_DEVICE); for (i = 0; i < ARRAY_SIZE(unmap_pool); i++) {
dma_dest = dma_map_page(dev->dev, page, offset, len, DMA_FROM_DEVICE); struct dmaengine_unmap_pool *p = &unmap_pool[i];
flags = DMA_CTRL_ACK | DMA_COMPL_SRC_UNMAP_SINGLE; size_t size;
tx = dev->device_prep_dma_memcpy(chan, dma_dest, dma_src, len, flags);
if (!tx) { size = sizeof(struct dmaengine_unmap_data) +
dma_unmap_single(dev->dev, dma_src, len, DMA_TO_DEVICE); sizeof(dma_addr_t) * p->size;
dma_unmap_page(dev->dev, dma_dest, len, DMA_FROM_DEVICE);
return -ENOMEM; p->cache = kmem_cache_create(p->name, size, 0,
SLAB_HWCACHE_ALIGN, NULL);
if (!p->cache)
break;
p->pool = mempool_create_slab_pool(1, p->cache);
if (!p->pool)
break;
} }
tx->callback = NULL; if (i == ARRAY_SIZE(unmap_pool))
cookie = tx->tx_submit(tx); return 0;
preempt_disable(); dmaengine_destroy_unmap_pool();
__this_cpu_add(chan->local->bytes_transferred, len); return -ENOMEM;
__this_cpu_inc(chan->local->memcpy_count); }
preempt_enable();
return cookie; struct dmaengine_unmap_data *
dmaengine_get_unmap_data(struct device *dev, int nr, gfp_t flags)
{
struct dmaengine_unmap_data *unmap;
unmap = mempool_alloc(__get_unmap_pool(nr)->pool, flags);
if (!unmap)
return NULL;
memset(unmap, 0, sizeof(*unmap));
kref_init(&unmap->kref);
unmap->dev = dev;
return unmap;
} }
EXPORT_SYMBOL(dma_async_memcpy_buf_to_pg); EXPORT_SYMBOL(dmaengine_get_unmap_data);
/** /**
* dma_async_memcpy_pg_to_pg - offloaded copy from page to page * dma_async_memcpy_pg_to_pg - offloaded copy from page to page
...@@ -1015,24 +1050,33 @@ dma_async_memcpy_pg_to_pg(struct dma_chan *chan, struct page *dest_pg, ...@@ -1015,24 +1050,33 @@ dma_async_memcpy_pg_to_pg(struct dma_chan *chan, struct page *dest_pg,
{ {
struct dma_device *dev = chan->device; struct dma_device *dev = chan->device;
struct dma_async_tx_descriptor *tx; struct dma_async_tx_descriptor *tx;
dma_addr_t dma_dest, dma_src; struct dmaengine_unmap_data *unmap;
dma_cookie_t cookie; dma_cookie_t cookie;
unsigned long flags; unsigned long flags;
dma_src = dma_map_page(dev->dev, src_pg, src_off, len, DMA_TO_DEVICE); unmap = dmaengine_get_unmap_data(dev->dev, 2, GFP_NOIO);
dma_dest = dma_map_page(dev->dev, dest_pg, dest_off, len, if (!unmap)
DMA_FROM_DEVICE); return -ENOMEM;
unmap->to_cnt = 1;
unmap->from_cnt = 1;
unmap->addr[0] = dma_map_page(dev->dev, src_pg, src_off, len,
DMA_TO_DEVICE);
unmap->addr[1] = dma_map_page(dev->dev, dest_pg, dest_off, len,
DMA_FROM_DEVICE);
unmap->len = len;
flags = DMA_CTRL_ACK; flags = DMA_CTRL_ACK;
tx = dev->device_prep_dma_memcpy(chan, dma_dest, dma_src, len, flags); tx = dev->device_prep_dma_memcpy(chan, unmap->addr[1], unmap->addr[0],
len, flags);
if (!tx) { if (!tx) {
dma_unmap_page(dev->dev, dma_src, len, DMA_TO_DEVICE); dmaengine_unmap_put(unmap);
dma_unmap_page(dev->dev, dma_dest, len, DMA_FROM_DEVICE);
return -ENOMEM; return -ENOMEM;
} }
tx->callback = NULL; dma_set_unmap(tx, unmap);
cookie = tx->tx_submit(tx); cookie = tx->tx_submit(tx);
dmaengine_unmap_put(unmap);
preempt_disable(); preempt_disable();
__this_cpu_add(chan->local->bytes_transferred, len); __this_cpu_add(chan->local->bytes_transferred, len);
...@@ -1043,6 +1087,52 @@ dma_async_memcpy_pg_to_pg(struct dma_chan *chan, struct page *dest_pg, ...@@ -1043,6 +1087,52 @@ dma_async_memcpy_pg_to_pg(struct dma_chan *chan, struct page *dest_pg,
} }
EXPORT_SYMBOL(dma_async_memcpy_pg_to_pg); EXPORT_SYMBOL(dma_async_memcpy_pg_to_pg);
/**
* dma_async_memcpy_buf_to_buf - offloaded copy between virtual addresses
* @chan: DMA channel to offload copy to
* @dest: destination address (virtual)
* @src: source address (virtual)
* @len: length
*
* Both @dest and @src must be mappable to a bus address according to the
* DMA mapping API rules for streaming mappings.
* Both @dest and @src must stay memory resident (kernel memory or locked
* user space pages).
*/
dma_cookie_t
dma_async_memcpy_buf_to_buf(struct dma_chan *chan, void *dest,
void *src, size_t len)
{
return dma_async_memcpy_pg_to_pg(chan, virt_to_page(dest),
(unsigned long) dest & ~PAGE_MASK,
virt_to_page(src),
(unsigned long) src & ~PAGE_MASK, len);
}
EXPORT_SYMBOL(dma_async_memcpy_buf_to_buf);
/**
* dma_async_memcpy_buf_to_pg - offloaded copy from address to page
* @chan: DMA channel to offload copy to
* @page: destination page
* @offset: offset in page to copy to
* @kdata: source address (virtual)
* @len: length
*
* Both @page/@offset and @kdata must be mappable to a bus address according
* to the DMA mapping API rules for streaming mappings.
* Both @page/@offset and @kdata must stay memory resident (kernel memory or
* locked user space pages)
*/
dma_cookie_t
dma_async_memcpy_buf_to_pg(struct dma_chan *chan, struct page *page,
unsigned int offset, void *kdata, size_t len)
{
return dma_async_memcpy_pg_to_pg(chan, page, offset,
virt_to_page(kdata),
(unsigned long) kdata & ~PAGE_MASK, len);
}
EXPORT_SYMBOL(dma_async_memcpy_buf_to_pg);
void dma_async_tx_descriptor_init(struct dma_async_tx_descriptor *tx, void dma_async_tx_descriptor_init(struct dma_async_tx_descriptor *tx,
struct dma_chan *chan) struct dma_chan *chan)
{ {
...@@ -1062,7 +1152,7 @@ dma_wait_for_async_tx(struct dma_async_tx_descriptor *tx) ...@@ -1062,7 +1152,7 @@ dma_wait_for_async_tx(struct dma_async_tx_descriptor *tx)
unsigned long dma_sync_wait_timeout = jiffies + msecs_to_jiffies(5000); unsigned long dma_sync_wait_timeout = jiffies + msecs_to_jiffies(5000);
if (!tx) if (!tx)
return DMA_SUCCESS; return DMA_COMPLETE;
while (tx->cookie == -EBUSY) { while (tx->cookie == -EBUSY) {
if (time_after_eq(jiffies, dma_sync_wait_timeout)) { if (time_after_eq(jiffies, dma_sync_wait_timeout)) {
...@@ -1116,6 +1206,10 @@ EXPORT_SYMBOL_GPL(dma_run_dependencies); ...@@ -1116,6 +1206,10 @@ EXPORT_SYMBOL_GPL(dma_run_dependencies);
static int __init dma_bus_init(void) static int __init dma_bus_init(void)
{ {
int err = dmaengine_init_unmap_pool();
if (err)
return err;
return class_register(&dma_devclass); return class_register(&dma_devclass);
} }
arch_initcall(dma_bus_init); arch_initcall(dma_bus_init);
......
This diff is collapsed.
...@@ -85,10 +85,6 @@ static struct device *chan2dev(struct dma_chan *chan) ...@@ -85,10 +85,6 @@ static struct device *chan2dev(struct dma_chan *chan)
{ {
return &chan->dev->device; return &chan->dev->device;
} }
static struct device *chan2parent(struct dma_chan *chan)
{
return chan->dev->device.parent;
}
static struct dw_desc *dwc_first_active(struct dw_dma_chan *dwc) static struct dw_desc *dwc_first_active(struct dw_dma_chan *dwc)
{ {
...@@ -311,26 +307,7 @@ dwc_descriptor_complete(struct dw_dma_chan *dwc, struct dw_desc *desc, ...@@ -311,26 +307,7 @@ dwc_descriptor_complete(struct dw_dma_chan *dwc, struct dw_desc *desc,
list_splice_init(&desc->tx_list, &dwc->free_list); list_splice_init(&desc->tx_list, &dwc->free_list);
list_move(&desc->desc_node, &dwc->free_list); list_move(&desc->desc_node, &dwc->free_list);
if (!is_slave_direction(dwc->direction)) { dma_descriptor_unmap(txd);
struct device *parent = chan2parent(&dwc->chan);
if (!(txd->flags & DMA_COMPL_SKIP_DEST_UNMAP)) {
if (txd->flags & DMA_COMPL_DEST_UNMAP_SINGLE)
dma_unmap_single(parent, desc->lli.dar,
desc->total_len, DMA_FROM_DEVICE);
else
dma_unmap_page(parent, desc->lli.dar,
desc->total_len, DMA_FROM_DEVICE);
}
if (!(txd->flags & DMA_COMPL_SKIP_SRC_UNMAP)) {
if (txd->flags & DMA_COMPL_SRC_UNMAP_SINGLE)
dma_unmap_single(parent, desc->lli.sar,
desc->total_len, DMA_TO_DEVICE);
else
dma_unmap_page(parent, desc->lli.sar,
desc->total_len, DMA_TO_DEVICE);
}
}
spin_unlock_irqrestore(&dwc->lock, flags); spin_unlock_irqrestore(&dwc->lock, flags);
if (callback) if (callback)
...@@ -1098,13 +1075,13 @@ dwc_tx_status(struct dma_chan *chan, ...@@ -1098,13 +1075,13 @@ dwc_tx_status(struct dma_chan *chan,
enum dma_status ret; enum dma_status ret;
ret = dma_cookie_status(chan, cookie, txstate); ret = dma_cookie_status(chan, cookie, txstate);
if (ret == DMA_SUCCESS) if (ret == DMA_COMPLETE)
return ret; return ret;
dwc_scan_descriptors(to_dw_dma(chan->device), dwc); dwc_scan_descriptors(to_dw_dma(chan->device), dwc);
ret = dma_cookie_status(chan, cookie, txstate); ret = dma_cookie_status(chan, cookie, txstate);
if (ret != DMA_SUCCESS) if (ret != DMA_COMPLETE)
dma_set_residue(txstate, dwc_get_residue(dwc)); dma_set_residue(txstate, dwc_get_residue(dwc));
if (dwc->paused && ret == DMA_IN_PROGRESS) if (dwc->paused && ret == DMA_IN_PROGRESS)
......
This diff is collapsed.
...@@ -733,28 +733,6 @@ static void ep93xx_dma_advance_work(struct ep93xx_dma_chan *edmac) ...@@ -733,28 +733,6 @@ static void ep93xx_dma_advance_work(struct ep93xx_dma_chan *edmac)
spin_unlock_irqrestore(&edmac->lock, flags); spin_unlock_irqrestore(&edmac->lock, flags);
} }
static void ep93xx_dma_unmap_buffers(struct ep93xx_dma_desc *desc)
{
struct device *dev = desc->txd.chan->device->dev;
if (!(desc->txd.flags & DMA_COMPL_SKIP_SRC_UNMAP)) {
if (desc->txd.flags & DMA_COMPL_SRC_UNMAP_SINGLE)
dma_unmap_single(dev, desc->src_addr, desc->size,
DMA_TO_DEVICE);
else
dma_unmap_page(dev, desc->src_addr, desc->size,
DMA_TO_DEVICE);
}
if (!(desc->txd.flags & DMA_COMPL_SKIP_DEST_UNMAP)) {
if (desc->txd.flags & DMA_COMPL_DEST_UNMAP_SINGLE)
dma_unmap_single(dev, desc->dst_addr, desc->size,
DMA_FROM_DEVICE);
else
dma_unmap_page(dev, desc->dst_addr, desc->size,
DMA_FROM_DEVICE);
}
}
static void ep93xx_dma_tasklet(unsigned long data) static void ep93xx_dma_tasklet(unsigned long data)
{ {
struct ep93xx_dma_chan *edmac = (struct ep93xx_dma_chan *)data; struct ep93xx_dma_chan *edmac = (struct ep93xx_dma_chan *)data;
...@@ -787,13 +765,7 @@ static void ep93xx_dma_tasklet(unsigned long data) ...@@ -787,13 +765,7 @@ static void ep93xx_dma_tasklet(unsigned long data)
/* Now we can release all the chained descriptors */ /* Now we can release all the chained descriptors */
list_for_each_entry_safe(desc, d, &list, node) { list_for_each_entry_safe(desc, d, &list, node) {
/* dma_descriptor_unmap(&desc->txd);
* For the memcpy channels the API requires us to unmap the
* buffers unless requested otherwise.
*/
if (!edmac->chan.private)
ep93xx_dma_unmap_buffers(desc);
ep93xx_dma_desc_put(edmac, desc); ep93xx_dma_desc_put(edmac, desc);
} }
......
...@@ -870,22 +870,7 @@ static void fsldma_cleanup_descriptor(struct fsldma_chan *chan, ...@@ -870,22 +870,7 @@ static void fsldma_cleanup_descriptor(struct fsldma_chan *chan,
/* Run any dependencies */ /* Run any dependencies */
dma_run_dependencies(txd); dma_run_dependencies(txd);
/* Unmap the dst buffer, if requested */ dma_descriptor_unmap(txd);
if (!(txd->flags & DMA_COMPL_SKIP_DEST_UNMAP)) {
if (txd->flags & DMA_COMPL_DEST_UNMAP_SINGLE)
dma_unmap_single(dev, dst, len, DMA_FROM_DEVICE);
else
dma_unmap_page(dev, dst, len, DMA_FROM_DEVICE);
}
/* Unmap the src buffer, if requested */
if (!(txd->flags & DMA_COMPL_SKIP_SRC_UNMAP)) {
if (txd->flags & DMA_COMPL_SRC_UNMAP_SINGLE)
dma_unmap_single(dev, src, len, DMA_TO_DEVICE);
else
dma_unmap_page(dev, src, len, DMA_TO_DEVICE);
}
#ifdef FSL_DMA_LD_DEBUG #ifdef FSL_DMA_LD_DEBUG
chan_dbg(chan, "LD %p free\n", desc); chan_dbg(chan, "LD %p free\n", desc);
#endif #endif
...@@ -1255,7 +1240,9 @@ static int fsl_dma_chan_probe(struct fsldma_device *fdev, ...@@ -1255,7 +1240,9 @@ static int fsl_dma_chan_probe(struct fsldma_device *fdev,
WARN_ON(fdev->feature != chan->feature); WARN_ON(fdev->feature != chan->feature);
chan->dev = fdev->dev; chan->dev = fdev->dev;
chan->id = ((res.start - 0x100) & 0xfff) >> 7; chan->id = (res.start & 0xfff) < 0x300 ?
((res.start - 0x100) & 0xfff) >> 7 :
((res.start - 0x200) & 0xfff) >> 7;
if (chan->id >= FSL_DMA_MAX_CHANS_PER_DEVICE) { if (chan->id >= FSL_DMA_MAX_CHANS_PER_DEVICE) {
dev_err(fdev->dev, "too many channels for device\n"); dev_err(fdev->dev, "too many channels for device\n");
err = -EINVAL; err = -EINVAL;
...@@ -1428,6 +1415,7 @@ static int fsldma_of_remove(struct platform_device *op) ...@@ -1428,6 +1415,7 @@ static int fsldma_of_remove(struct platform_device *op)
} }
static const struct of_device_id fsldma_of_ids[] = { static const struct of_device_id fsldma_of_ids[] = {
{ .compatible = "fsl,elo3-dma", },
{ .compatible = "fsl,eloplus-dma", }, { .compatible = "fsl,eloplus-dma", },
{ .compatible = "fsl,elo-dma", }, { .compatible = "fsl,elo-dma", },
{} {}
...@@ -1449,7 +1437,7 @@ static struct platform_driver fsldma_of_driver = { ...@@ -1449,7 +1437,7 @@ static struct platform_driver fsldma_of_driver = {
static __init int fsldma_init(void) static __init int fsldma_init(void)
{ {
pr_info("Freescale Elo / Elo Plus DMA driver\n"); pr_info("Freescale Elo series DMA driver\n");
return platform_driver_register(&fsldma_of_driver); return platform_driver_register(&fsldma_of_driver);
} }
...@@ -1461,5 +1449,5 @@ static void __exit fsldma_exit(void) ...@@ -1461,5 +1449,5 @@ static void __exit fsldma_exit(void)
subsys_initcall(fsldma_init); subsys_initcall(fsldma_init);
module_exit(fsldma_exit); module_exit(fsldma_exit);
MODULE_DESCRIPTION("Freescale Elo / Elo Plus DMA driver"); MODULE_DESCRIPTION("Freescale Elo series DMA driver");
MODULE_LICENSE("GPL"); MODULE_LICENSE("GPL");
...@@ -112,7 +112,7 @@ struct fsldma_chan_regs { ...@@ -112,7 +112,7 @@ struct fsldma_chan_regs {
}; };
struct fsldma_chan; struct fsldma_chan;
#define FSL_DMA_MAX_CHANS_PER_DEVICE 4 #define FSL_DMA_MAX_CHANS_PER_DEVICE 8
struct fsldma_device { struct fsldma_device {
void __iomem *regs; /* DGSR register base */ void __iomem *regs; /* DGSR register base */
......
...@@ -572,9 +572,11 @@ static int imxdma_xfer_desc(struct imxdma_desc *d) ...@@ -572,9 +572,11 @@ static int imxdma_xfer_desc(struct imxdma_desc *d)
imx_dmav1_writel(imxdma, d->len, DMA_CNTR(imxdmac->channel)); imx_dmav1_writel(imxdma, d->len, DMA_CNTR(imxdmac->channel));
dev_dbg(imxdma->dev, "%s channel: %d dest=0x%08x src=0x%08x " dev_dbg(imxdma->dev,
"dma_length=%d\n", __func__, imxdmac->channel, "%s channel: %d dest=0x%08llx src=0x%08llx dma_length=%zu\n",
d->dest, d->src, d->len); __func__, imxdmac->channel,
(unsigned long long)d->dest,
(unsigned long long)d->src, d->len);
break; break;
/* Cyclic transfer is the same as slave_sg with special sg configuration. */ /* Cyclic transfer is the same as slave_sg with special sg configuration. */
...@@ -586,20 +588,22 @@ static int imxdma_xfer_desc(struct imxdma_desc *d) ...@@ -586,20 +588,22 @@ static int imxdma_xfer_desc(struct imxdma_desc *d)
imx_dmav1_writel(imxdma, imxdmac->ccr_from_device, imx_dmav1_writel(imxdma, imxdmac->ccr_from_device,
DMA_CCR(imxdmac->channel)); DMA_CCR(imxdmac->channel));
dev_dbg(imxdma->dev, "%s channel: %d sg=%p sgcount=%d " dev_dbg(imxdma->dev,
"total length=%d dev_addr=0x%08x (dev2mem)\n", "%s channel: %d sg=%p sgcount=%d total length=%zu dev_addr=0x%08llx (dev2mem)\n",
__func__, imxdmac->channel, d->sg, d->sgcount, __func__, imxdmac->channel,
d->len, imxdmac->per_address); d->sg, d->sgcount, d->len,
(unsigned long long)imxdmac->per_address);
} else if (d->direction == DMA_MEM_TO_DEV) { } else if (d->direction == DMA_MEM_TO_DEV) {
imx_dmav1_writel(imxdma, imxdmac->per_address, imx_dmav1_writel(imxdma, imxdmac->per_address,
DMA_DAR(imxdmac->channel)); DMA_DAR(imxdmac->channel));
imx_dmav1_writel(imxdma, imxdmac->ccr_to_device, imx_dmav1_writel(imxdma, imxdmac->ccr_to_device,
DMA_CCR(imxdmac->channel)); DMA_CCR(imxdmac->channel));
dev_dbg(imxdma->dev, "%s channel: %d sg=%p sgcount=%d " dev_dbg(imxdma->dev,
"total length=%d dev_addr=0x%08x (mem2dev)\n", "%s channel: %d sg=%p sgcount=%d total length=%zu dev_addr=0x%08llx (mem2dev)\n",
__func__, imxdmac->channel, d->sg, d->sgcount, __func__, imxdmac->channel,
d->len, imxdmac->per_address); d->sg, d->sgcount, d->len,
(unsigned long long)imxdmac->per_address);
} else { } else {
dev_err(imxdma->dev, "%s channel: %d bad dma mode\n", dev_err(imxdma->dev, "%s channel: %d bad dma mode\n",
__func__, imxdmac->channel); __func__, imxdmac->channel);
...@@ -771,7 +775,7 @@ static int imxdma_alloc_chan_resources(struct dma_chan *chan) ...@@ -771,7 +775,7 @@ static int imxdma_alloc_chan_resources(struct dma_chan *chan)
desc->desc.tx_submit = imxdma_tx_submit; desc->desc.tx_submit = imxdma_tx_submit;
/* txd.flags will be overwritten in prep funcs */ /* txd.flags will be overwritten in prep funcs */
desc->desc.flags = DMA_CTRL_ACK; desc->desc.flags = DMA_CTRL_ACK;
desc->status = DMA_SUCCESS; desc->status = DMA_COMPLETE;
list_add_tail(&desc->node, &imxdmac->ld_free); list_add_tail(&desc->node, &imxdmac->ld_free);
imxdmac->descs_allocated++; imxdmac->descs_allocated++;
...@@ -870,7 +874,7 @@ static struct dma_async_tx_descriptor *imxdma_prep_dma_cyclic( ...@@ -870,7 +874,7 @@ static struct dma_async_tx_descriptor *imxdma_prep_dma_cyclic(
int i; int i;
unsigned int periods = buf_len / period_len; unsigned int periods = buf_len / period_len;
dev_dbg(imxdma->dev, "%s channel: %d buf_len=%d period_len=%d\n", dev_dbg(imxdma->dev, "%s channel: %d buf_len=%zu period_len=%zu\n",
__func__, imxdmac->channel, buf_len, period_len); __func__, imxdmac->channel, buf_len, period_len);
if (list_empty(&imxdmac->ld_free) || if (list_empty(&imxdmac->ld_free) ||
...@@ -926,8 +930,9 @@ static struct dma_async_tx_descriptor *imxdma_prep_dma_memcpy( ...@@ -926,8 +930,9 @@ static struct dma_async_tx_descriptor *imxdma_prep_dma_memcpy(
struct imxdma_engine *imxdma = imxdmac->imxdma; struct imxdma_engine *imxdma = imxdmac->imxdma;
struct imxdma_desc *desc; struct imxdma_desc *desc;
dev_dbg(imxdma->dev, "%s channel: %d src=0x%x dst=0x%x len=%d\n", dev_dbg(imxdma->dev, "%s channel: %d src=0x%llx dst=0x%llx len=%zu\n",
__func__, imxdmac->channel, src, dest, len); __func__, imxdmac->channel, (unsigned long long)src,
(unsigned long long)dest, len);
if (list_empty(&imxdmac->ld_free) || if (list_empty(&imxdmac->ld_free) ||
imxdma_chan_is_doing_cyclic(imxdmac)) imxdma_chan_is_doing_cyclic(imxdmac))
...@@ -956,9 +961,10 @@ static struct dma_async_tx_descriptor *imxdma_prep_dma_interleaved( ...@@ -956,9 +961,10 @@ static struct dma_async_tx_descriptor *imxdma_prep_dma_interleaved(
struct imxdma_engine *imxdma = imxdmac->imxdma; struct imxdma_engine *imxdma = imxdmac->imxdma;
struct imxdma_desc *desc; struct imxdma_desc *desc;
dev_dbg(imxdma->dev, "%s channel: %d src_start=0x%x dst_start=0x%x\n" dev_dbg(imxdma->dev, "%s channel: %d src_start=0x%llx dst_start=0x%llx\n"
" src_sgl=%s dst_sgl=%s numf=%d frame_size=%d\n", __func__, " src_sgl=%s dst_sgl=%s numf=%zu frame_size=%zu\n", __func__,
imxdmac->channel, xt->src_start, xt->dst_start, imxdmac->channel, (unsigned long long)xt->src_start,
(unsigned long long) xt->dst_start,
xt->src_sgl ? "true" : "false", xt->dst_sgl ? "true" : "false", xt->src_sgl ? "true" : "false", xt->dst_sgl ? "true" : "false",
xt->numf, xt->frame_size); xt->numf, xt->frame_size);
......
...@@ -638,7 +638,7 @@ static void mxc_sdma_handle_channel_normal(struct sdma_channel *sdmac) ...@@ -638,7 +638,7 @@ static void mxc_sdma_handle_channel_normal(struct sdma_channel *sdmac)
if (error) if (error)
sdmac->status = DMA_ERROR; sdmac->status = DMA_ERROR;
else else
sdmac->status = DMA_SUCCESS; sdmac->status = DMA_COMPLETE;
dma_cookie_complete(&sdmac->desc); dma_cookie_complete(&sdmac->desc);
if (sdmac->desc.callback) if (sdmac->desc.callback)
...@@ -1089,8 +1089,8 @@ static struct dma_async_tx_descriptor *sdma_prep_slave_sg( ...@@ -1089,8 +1089,8 @@ static struct dma_async_tx_descriptor *sdma_prep_slave_sg(
param &= ~BD_CONT; param &= ~BD_CONT;
} }
dev_dbg(sdma->dev, "entry %d: count: %d dma: 0x%08x %s%s\n", dev_dbg(sdma->dev, "entry %d: count: %d dma: %#llx %s%s\n",
i, count, sg->dma_address, i, count, (u64)sg->dma_address,
param & BD_WRAP ? "wrap" : "", param & BD_WRAP ? "wrap" : "",
param & BD_INTR ? " intr" : ""); param & BD_INTR ? " intr" : "");
...@@ -1163,8 +1163,8 @@ static struct dma_async_tx_descriptor *sdma_prep_dma_cyclic( ...@@ -1163,8 +1163,8 @@ static struct dma_async_tx_descriptor *sdma_prep_dma_cyclic(
if (i + 1 == num_periods) if (i + 1 == num_periods)
param |= BD_WRAP; param |= BD_WRAP;
dev_dbg(sdma->dev, "entry %d: count: %d dma: 0x%08x %s%s\n", dev_dbg(sdma->dev, "entry %d: count: %d dma: %#llx %s%s\n",
i, period_len, dma_addr, i, period_len, (u64)dma_addr,
param & BD_WRAP ? "wrap" : "", param & BD_WRAP ? "wrap" : "",
param & BD_INTR ? " intr" : ""); param & BD_INTR ? " intr" : "");
......
...@@ -309,7 +309,7 @@ static void midc_descriptor_complete(struct intel_mid_dma_chan *midc, ...@@ -309,7 +309,7 @@ static void midc_descriptor_complete(struct intel_mid_dma_chan *midc,
callback_txd(param_txd); callback_txd(param_txd);
} }
if (midc->raw_tfr) { if (midc->raw_tfr) {
desc->status = DMA_SUCCESS; desc->status = DMA_COMPLETE;
if (desc->lli != NULL) { if (desc->lli != NULL) {
pci_pool_free(desc->lli_pool, desc->lli, pci_pool_free(desc->lli_pool, desc->lli,
desc->lli_phys); desc->lli_phys);
...@@ -481,7 +481,7 @@ static enum dma_status intel_mid_dma_tx_status(struct dma_chan *chan, ...@@ -481,7 +481,7 @@ static enum dma_status intel_mid_dma_tx_status(struct dma_chan *chan,
enum dma_status ret; enum dma_status ret;
ret = dma_cookie_status(chan, cookie, txstate); ret = dma_cookie_status(chan, cookie, txstate);
if (ret != DMA_SUCCESS) { if (ret != DMA_COMPLETE) {
spin_lock_bh(&midc->lock); spin_lock_bh(&midc->lock);
midc_scan_descriptors(to_middma_device(chan->device), midc); midc_scan_descriptors(to_middma_device(chan->device), midc);
spin_unlock_bh(&midc->lock); spin_unlock_bh(&midc->lock);
......
...@@ -531,21 +531,6 @@ static void ioat1_cleanup_event(unsigned long data) ...@@ -531,21 +531,6 @@ static void ioat1_cleanup_event(unsigned long data)
writew(IOAT_CHANCTRL_RUN, ioat->base.reg_base + IOAT_CHANCTRL_OFFSET); writew(IOAT_CHANCTRL_RUN, ioat->base.reg_base + IOAT_CHANCTRL_OFFSET);
} }
void ioat_dma_unmap(struct ioat_chan_common *chan, enum dma_ctrl_flags flags,
size_t len, struct ioat_dma_descriptor *hw)
{
struct pci_dev *pdev = chan->device->pdev;
size_t offset = len - hw->size;
if (!(flags & DMA_COMPL_SKIP_DEST_UNMAP))
ioat_unmap(pdev, hw->dst_addr - offset, len,
PCI_DMA_FROMDEVICE, flags, 1);
if (!(flags & DMA_COMPL_SKIP_SRC_UNMAP))
ioat_unmap(pdev, hw->src_addr - offset, len,
PCI_DMA_TODEVICE, flags, 0);
}
dma_addr_t ioat_get_current_completion(struct ioat_chan_common *chan) dma_addr_t ioat_get_current_completion(struct ioat_chan_common *chan)
{ {
dma_addr_t phys_complete; dma_addr_t phys_complete;
...@@ -602,7 +587,7 @@ static void __cleanup(struct ioat_dma_chan *ioat, dma_addr_t phys_complete) ...@@ -602,7 +587,7 @@ static void __cleanup(struct ioat_dma_chan *ioat, dma_addr_t phys_complete)
dump_desc_dbg(ioat, desc); dump_desc_dbg(ioat, desc);
if (tx->cookie) { if (tx->cookie) {
dma_cookie_complete(tx); dma_cookie_complete(tx);
ioat_dma_unmap(chan, tx->flags, desc->len, desc->hw); dma_descriptor_unmap(tx);
ioat->active -= desc->hw->tx_cnt; ioat->active -= desc->hw->tx_cnt;
if (tx->callback) { if (tx->callback) {
tx->callback(tx->callback_param); tx->callback(tx->callback_param);
...@@ -733,7 +718,7 @@ ioat_dma_tx_status(struct dma_chan *c, dma_cookie_t cookie, ...@@ -733,7 +718,7 @@ ioat_dma_tx_status(struct dma_chan *c, dma_cookie_t cookie,
enum dma_status ret; enum dma_status ret;
ret = dma_cookie_status(c, cookie, txstate); ret = dma_cookie_status(c, cookie, txstate);
if (ret == DMA_SUCCESS) if (ret == DMA_COMPLETE)
return ret; return ret;
device->cleanup_fn((unsigned long) c); device->cleanup_fn((unsigned long) c);
...@@ -833,8 +818,7 @@ int ioat_dma_self_test(struct ioatdma_device *device) ...@@ -833,8 +818,7 @@ int ioat_dma_self_test(struct ioatdma_device *device)
dma_src = dma_map_single(dev, src, IOAT_TEST_SIZE, DMA_TO_DEVICE); dma_src = dma_map_single(dev, src, IOAT_TEST_SIZE, DMA_TO_DEVICE);
dma_dest = dma_map_single(dev, dest, IOAT_TEST_SIZE, DMA_FROM_DEVICE); dma_dest = dma_map_single(dev, dest, IOAT_TEST_SIZE, DMA_FROM_DEVICE);
flags = DMA_COMPL_SKIP_SRC_UNMAP | DMA_COMPL_SKIP_DEST_UNMAP | flags = DMA_PREP_INTERRUPT;
DMA_PREP_INTERRUPT;
tx = device->common.device_prep_dma_memcpy(dma_chan, dma_dest, dma_src, tx = device->common.device_prep_dma_memcpy(dma_chan, dma_dest, dma_src,
IOAT_TEST_SIZE, flags); IOAT_TEST_SIZE, flags);
if (!tx) { if (!tx) {
...@@ -859,7 +843,7 @@ int ioat_dma_self_test(struct ioatdma_device *device) ...@@ -859,7 +843,7 @@ int ioat_dma_self_test(struct ioatdma_device *device)
if (tmo == 0 || if (tmo == 0 ||
dma->device_tx_status(dma_chan, cookie, NULL) dma->device_tx_status(dma_chan, cookie, NULL)
!= DMA_SUCCESS) { != DMA_COMPLETE) {
dev_err(dev, "Self-test copy timed out, disabling\n"); dev_err(dev, "Self-test copy timed out, disabling\n");
err = -ENODEV; err = -ENODEV;
goto unmap_dma; goto unmap_dma;
...@@ -885,8 +869,7 @@ static char ioat_interrupt_style[32] = "msix"; ...@@ -885,8 +869,7 @@ static char ioat_interrupt_style[32] = "msix";
module_param_string(ioat_interrupt_style, ioat_interrupt_style, module_param_string(ioat_interrupt_style, ioat_interrupt_style,
sizeof(ioat_interrupt_style), 0644); sizeof(ioat_interrupt_style), 0644);
MODULE_PARM_DESC(ioat_interrupt_style, MODULE_PARM_DESC(ioat_interrupt_style,
"set ioat interrupt style: msix (default), " "set ioat interrupt style: msix (default), msi, intx");
"msix-single-vector, msi, intx)");
/** /**
* ioat_dma_setup_interrupts - setup interrupt handler * ioat_dma_setup_interrupts - setup interrupt handler
...@@ -904,8 +887,6 @@ int ioat_dma_setup_interrupts(struct ioatdma_device *device) ...@@ -904,8 +887,6 @@ int ioat_dma_setup_interrupts(struct ioatdma_device *device)
if (!strcmp(ioat_interrupt_style, "msix")) if (!strcmp(ioat_interrupt_style, "msix"))
goto msix; goto msix;
if (!strcmp(ioat_interrupt_style, "msix-single-vector"))
goto msix_single_vector;
if (!strcmp(ioat_interrupt_style, "msi")) if (!strcmp(ioat_interrupt_style, "msi"))
goto msi; goto msi;
if (!strcmp(ioat_interrupt_style, "intx")) if (!strcmp(ioat_interrupt_style, "intx"))
...@@ -920,10 +901,8 @@ int ioat_dma_setup_interrupts(struct ioatdma_device *device) ...@@ -920,10 +901,8 @@ int ioat_dma_setup_interrupts(struct ioatdma_device *device)
device->msix_entries[i].entry = i; device->msix_entries[i].entry = i;
err = pci_enable_msix(pdev, device->msix_entries, msixcnt); err = pci_enable_msix(pdev, device->msix_entries, msixcnt);
if (err < 0) if (err)
goto msi; goto msi;
if (err > 0)
goto msix_single_vector;
for (i = 0; i < msixcnt; i++) { for (i = 0; i < msixcnt; i++) {
msix = &device->msix_entries[i]; msix = &device->msix_entries[i];
...@@ -937,29 +916,13 @@ int ioat_dma_setup_interrupts(struct ioatdma_device *device) ...@@ -937,29 +916,13 @@ int ioat_dma_setup_interrupts(struct ioatdma_device *device)
chan = ioat_chan_by_index(device, j); chan = ioat_chan_by_index(device, j);
devm_free_irq(dev, msix->vector, chan); devm_free_irq(dev, msix->vector, chan);
} }
goto msix_single_vector; goto msi;
} }
} }
intrctrl |= IOAT_INTRCTRL_MSIX_VECTOR_CONTROL; intrctrl |= IOAT_INTRCTRL_MSIX_VECTOR_CONTROL;
device->irq_mode = IOAT_MSIX; device->irq_mode = IOAT_MSIX;
goto done; goto done;
msix_single_vector:
msix = &device->msix_entries[0];
msix->entry = 0;
err = pci_enable_msix(pdev, device->msix_entries, 1);
if (err)
goto msi;
err = devm_request_irq(dev, msix->vector, ioat_dma_do_interrupt, 0,
"ioat-msix", device);
if (err) {
pci_disable_msix(pdev);
goto msi;
}
device->irq_mode = IOAT_MSIX_SINGLE;
goto done;
msi: msi:
err = pci_enable_msi(pdev); err = pci_enable_msi(pdev);
if (err) if (err)
...@@ -971,7 +934,7 @@ int ioat_dma_setup_interrupts(struct ioatdma_device *device) ...@@ -971,7 +934,7 @@ int ioat_dma_setup_interrupts(struct ioatdma_device *device)
pci_disable_msi(pdev); pci_disable_msi(pdev);
goto intx; goto intx;
} }
device->irq_mode = IOAT_MSIX; device->irq_mode = IOAT_MSI;
goto done; goto done;
intx: intx:
......
...@@ -52,7 +52,6 @@ ...@@ -52,7 +52,6 @@
enum ioat_irq_mode { enum ioat_irq_mode {
IOAT_NOIRQ = 0, IOAT_NOIRQ = 0,
IOAT_MSIX, IOAT_MSIX,
IOAT_MSIX_SINGLE,
IOAT_MSI, IOAT_MSI,
IOAT_INTX IOAT_INTX
}; };
...@@ -83,7 +82,6 @@ struct ioatdma_device { ...@@ -83,7 +82,6 @@ struct ioatdma_device {
struct pci_pool *completion_pool; struct pci_pool *completion_pool;
#define MAX_SED_POOLS 5 #define MAX_SED_POOLS 5
struct dma_pool *sed_hw_pool[MAX_SED_POOLS]; struct dma_pool *sed_hw_pool[MAX_SED_POOLS];
struct kmem_cache *sed_pool;
struct dma_device common; struct dma_device common;
u8 version; u8 version;
struct msix_entry msix_entries[4]; struct msix_entry msix_entries[4];
...@@ -342,16 +340,6 @@ static inline bool is_ioat_bug(unsigned long err) ...@@ -342,16 +340,6 @@ static inline bool is_ioat_bug(unsigned long err)
return !!err; return !!err;
} }
static inline void ioat_unmap(struct pci_dev *pdev, dma_addr_t addr, size_t len,
int direction, enum dma_ctrl_flags flags, bool dst)
{
if ((dst && (flags & DMA_COMPL_DEST_UNMAP_SINGLE)) ||
(!dst && (flags & DMA_COMPL_SRC_UNMAP_SINGLE)))
pci_unmap_single(pdev, addr, len, direction);
else
pci_unmap_page(pdev, addr, len, direction);
}
int ioat_probe(struct ioatdma_device *device); int ioat_probe(struct ioatdma_device *device);
int ioat_register(struct ioatdma_device *device); int ioat_register(struct ioatdma_device *device);
int ioat1_dma_probe(struct ioatdma_device *dev, int dca); int ioat1_dma_probe(struct ioatdma_device *dev, int dca);
...@@ -363,8 +351,6 @@ void ioat_init_channel(struct ioatdma_device *device, ...@@ -363,8 +351,6 @@ void ioat_init_channel(struct ioatdma_device *device,
struct ioat_chan_common *chan, int idx); struct ioat_chan_common *chan, int idx);
enum dma_status ioat_dma_tx_status(struct dma_chan *c, dma_cookie_t cookie, enum dma_status ioat_dma_tx_status(struct dma_chan *c, dma_cookie_t cookie,
struct dma_tx_state *txstate); struct dma_tx_state *txstate);
void ioat_dma_unmap(struct ioat_chan_common *chan, enum dma_ctrl_flags flags,
size_t len, struct ioat_dma_descriptor *hw);
bool ioat_cleanup_preamble(struct ioat_chan_common *chan, bool ioat_cleanup_preamble(struct ioat_chan_common *chan,
dma_addr_t *phys_complete); dma_addr_t *phys_complete);
void ioat_kobject_add(struct ioatdma_device *device, struct kobj_type *type); void ioat_kobject_add(struct ioatdma_device *device, struct kobj_type *type);
......
...@@ -148,7 +148,7 @@ static void __cleanup(struct ioat2_dma_chan *ioat, dma_addr_t phys_complete) ...@@ -148,7 +148,7 @@ static void __cleanup(struct ioat2_dma_chan *ioat, dma_addr_t phys_complete)
tx = &desc->txd; tx = &desc->txd;
dump_desc_dbg(ioat, desc); dump_desc_dbg(ioat, desc);
if (tx->cookie) { if (tx->cookie) {
ioat_dma_unmap(chan, tx->flags, desc->len, desc->hw); dma_descriptor_unmap(tx);
dma_cookie_complete(tx); dma_cookie_complete(tx);
if (tx->callback) { if (tx->callback) {
tx->callback(tx->callback_param); tx->callback(tx->callback_param);
......
...@@ -157,7 +157,6 @@ static inline void ioat2_set_chainaddr(struct ioat2_dma_chan *ioat, u64 addr) ...@@ -157,7 +157,6 @@ static inline void ioat2_set_chainaddr(struct ioat2_dma_chan *ioat, u64 addr)
int ioat2_dma_probe(struct ioatdma_device *dev, int dca); int ioat2_dma_probe(struct ioatdma_device *dev, int dca);
int ioat3_dma_probe(struct ioatdma_device *dev, int dca); int ioat3_dma_probe(struct ioatdma_device *dev, int dca);
void ioat3_dma_remove(struct ioatdma_device *dev);
struct dca_provider *ioat2_dca_init(struct pci_dev *pdev, void __iomem *iobase); struct dca_provider *ioat2_dca_init(struct pci_dev *pdev, void __iomem *iobase);
struct dca_provider *ioat3_dca_init(struct pci_dev *pdev, void __iomem *iobase); struct dca_provider *ioat3_dca_init(struct pci_dev *pdev, void __iomem *iobase);
int ioat2_check_space_lock(struct ioat2_dma_chan *ioat, int num_descs); int ioat2_check_space_lock(struct ioat2_dma_chan *ioat, int num_descs);
......
This diff is collapsed.
...@@ -123,6 +123,7 @@ module_param(ioat_dca_enabled, int, 0644); ...@@ -123,6 +123,7 @@ module_param(ioat_dca_enabled, int, 0644);
MODULE_PARM_DESC(ioat_dca_enabled, "control support of dca service (default: 1)"); MODULE_PARM_DESC(ioat_dca_enabled, "control support of dca service (default: 1)");
struct kmem_cache *ioat2_cache; struct kmem_cache *ioat2_cache;
struct kmem_cache *ioat3_sed_cache;
#define DRV_NAME "ioatdma" #define DRV_NAME "ioatdma"
...@@ -207,9 +208,6 @@ static void ioat_remove(struct pci_dev *pdev) ...@@ -207,9 +208,6 @@ static void ioat_remove(struct pci_dev *pdev)
if (!device) if (!device)
return; return;
if (device->version >= IOAT_VER_3_0)
ioat3_dma_remove(device);
dev_err(&pdev->dev, "Removing dma and dca services\n"); dev_err(&pdev->dev, "Removing dma and dca services\n");
if (device->dca) { if (device->dca) {
unregister_dca_provider(device->dca, &pdev->dev); unregister_dca_provider(device->dca, &pdev->dev);
...@@ -221,7 +219,7 @@ static void ioat_remove(struct pci_dev *pdev) ...@@ -221,7 +219,7 @@ static void ioat_remove(struct pci_dev *pdev)
static int __init ioat_init_module(void) static int __init ioat_init_module(void)
{ {
int err; int err = -ENOMEM;
pr_info("%s: Intel(R) QuickData Technology Driver %s\n", pr_info("%s: Intel(R) QuickData Technology Driver %s\n",
DRV_NAME, IOAT_DMA_VERSION); DRV_NAME, IOAT_DMA_VERSION);
...@@ -231,9 +229,21 @@ static int __init ioat_init_module(void) ...@@ -231,9 +229,21 @@ static int __init ioat_init_module(void)
if (!ioat2_cache) if (!ioat2_cache)
return -ENOMEM; return -ENOMEM;
ioat3_sed_cache = KMEM_CACHE(ioat_sed_ent, 0);
if (!ioat3_sed_cache)
goto err_ioat2_cache;
err = pci_register_driver(&ioat_pci_driver); err = pci_register_driver(&ioat_pci_driver);
if (err) if (err)
kmem_cache_destroy(ioat2_cache); goto err_ioat3_cache;
return 0;
err_ioat3_cache:
kmem_cache_destroy(ioat3_sed_cache);
err_ioat2_cache:
kmem_cache_destroy(ioat2_cache);
return err; return err;
} }
......
This diff is collapsed.
...@@ -1232,8 +1232,10 @@ static irqreturn_t idmac_interrupt(int irq, void *dev_id) ...@@ -1232,8 +1232,10 @@ static irqreturn_t idmac_interrupt(int irq, void *dev_id)
desc = list_entry(ichan->queue.next, struct idmac_tx_desc, list); desc = list_entry(ichan->queue.next, struct idmac_tx_desc, list);
descnew = desc; descnew = desc;
dev_dbg(dev, "IDMAC irq %d, dma 0x%08x, next dma 0x%08x, current %d, curbuf 0x%08x\n", dev_dbg(dev, "IDMAC irq %d, dma %#llx, next dma %#llx, current %d, curbuf %#x\n",
irq, sg_dma_address(*sg), sgnext ? sg_dma_address(sgnext) : 0, ichan->active_buffer, curbuf); irq, (u64)sg_dma_address(*sg),
sgnext ? (u64)sg_dma_address(sgnext) : 0,
ichan->active_buffer, curbuf);
/* Find the descriptor of sgnext */ /* Find the descriptor of sgnext */
sgnew = idmac_sg_next(ichan, &descnew, *sg); sgnew = idmac_sg_next(ichan, &descnew, *sg);
......
...@@ -344,7 +344,7 @@ static enum dma_status k3_dma_tx_status(struct dma_chan *chan, ...@@ -344,7 +344,7 @@ static enum dma_status k3_dma_tx_status(struct dma_chan *chan,
size_t bytes = 0; size_t bytes = 0;
ret = dma_cookie_status(&c->vc.chan, cookie, state); ret = dma_cookie_status(&c->vc.chan, cookie, state);
if (ret == DMA_SUCCESS) if (ret == DMA_COMPLETE)
return ret; return ret;
spin_lock_irqsave(&c->vc.lock, flags); spin_lock_irqsave(&c->vc.lock, flags);
...@@ -693,7 +693,7 @@ static int k3_dma_probe(struct platform_device *op) ...@@ -693,7 +693,7 @@ static int k3_dma_probe(struct platform_device *op)
irq = platform_get_irq(op, 0); irq = platform_get_irq(op, 0);
ret = devm_request_irq(&op->dev, irq, ret = devm_request_irq(&op->dev, irq,
k3_dma_int_handler, IRQF_DISABLED, DRIVER_NAME, d); k3_dma_int_handler, 0, DRIVER_NAME, d);
if (ret) if (ret)
return ret; return ret;
......
...@@ -798,8 +798,7 @@ static void dma_do_tasklet(unsigned long data) ...@@ -798,8 +798,7 @@ static void dma_do_tasklet(unsigned long data)
* move the descriptors to a temporary list so we can drop * move the descriptors to a temporary list so we can drop
* the lock during the entire cleanup operation * the lock during the entire cleanup operation
*/ */
list_del(&desc->node); list_move(&desc->node, &chain_cleanup);
list_add(&desc->node, &chain_cleanup);
/* /*
* Look for the first list entry which has the ENDIRQEN flag * Look for the first list entry which has the ENDIRQEN flag
...@@ -863,7 +862,7 @@ static int mmp_pdma_chan_init(struct mmp_pdma_device *pdev, ...@@ -863,7 +862,7 @@ static int mmp_pdma_chan_init(struct mmp_pdma_device *pdev,
if (irq) { if (irq) {
ret = devm_request_irq(pdev->dev, irq, ret = devm_request_irq(pdev->dev, irq,
mmp_pdma_chan_handler, IRQF_DISABLED, "pdma", phy); mmp_pdma_chan_handler, 0, "pdma", phy);
if (ret) { if (ret) {
dev_err(pdev->dev, "channel request irq fail!\n"); dev_err(pdev->dev, "channel request irq fail!\n");
return ret; return ret;
...@@ -970,7 +969,7 @@ static int mmp_pdma_probe(struct platform_device *op) ...@@ -970,7 +969,7 @@ static int mmp_pdma_probe(struct platform_device *op)
/* all chan share one irq, demux inside */ /* all chan share one irq, demux inside */
irq = platform_get_irq(op, 0); irq = platform_get_irq(op, 0);
ret = devm_request_irq(pdev->dev, irq, ret = devm_request_irq(pdev->dev, irq,
mmp_pdma_int_handler, IRQF_DISABLED, "pdma", pdev); mmp_pdma_int_handler, 0, "pdma", pdev);
if (ret) if (ret)
return ret; return ret;
} }
......
This diff is collapsed.
This diff is collapsed.
...@@ -34,13 +34,13 @@ ...@@ -34,13 +34,13 @@
#define XOR_OPERATION_MODE_MEMCPY 2 #define XOR_OPERATION_MODE_MEMCPY 2
#define XOR_DESCRIPTOR_SWAP BIT(14) #define XOR_DESCRIPTOR_SWAP BIT(14)
#define XOR_CURR_DESC(chan) (chan->mmr_base + 0x210 + (chan->idx * 4)) #define XOR_CURR_DESC(chan) (chan->mmr_high_base + 0x10 + (chan->idx * 4))
#define XOR_NEXT_DESC(chan) (chan->mmr_base + 0x200 + (chan->idx * 4)) #define XOR_NEXT_DESC(chan) (chan->mmr_high_base + 0x00 + (chan->idx * 4))
#define XOR_BYTE_COUNT(chan) (chan->mmr_base + 0x220 + (chan->idx * 4)) #define XOR_BYTE_COUNT(chan) (chan->mmr_high_base + 0x20 + (chan->idx * 4))
#define XOR_DEST_POINTER(chan) (chan->mmr_base + 0x2B0 + (chan->idx * 4)) #define XOR_DEST_POINTER(chan) (chan->mmr_high_base + 0xB0 + (chan->idx * 4))
#define XOR_BLOCK_SIZE(chan) (chan->mmr_base + 0x2C0 + (chan->idx * 4)) #define XOR_BLOCK_SIZE(chan) (chan->mmr_high_base + 0xC0 + (chan->idx * 4))
#define XOR_INIT_VALUE_LOW(chan) (chan->mmr_base + 0x2E0) #define XOR_INIT_VALUE_LOW(chan) (chan->mmr_high_base + 0xE0)
#define XOR_INIT_VALUE_HIGH(chan) (chan->mmr_base + 0x2E4) #define XOR_INIT_VALUE_HIGH(chan) (chan->mmr_high_base + 0xE4)
#define XOR_CONFIG(chan) (chan->mmr_base + 0x10 + (chan->idx * 4)) #define XOR_CONFIG(chan) (chan->mmr_base + 0x10 + (chan->idx * 4))
#define XOR_ACTIVATION(chan) (chan->mmr_base + 0x20 + (chan->idx * 4)) #define XOR_ACTIVATION(chan) (chan->mmr_base + 0x20 + (chan->idx * 4))
...@@ -50,11 +50,11 @@ ...@@ -50,11 +50,11 @@
#define XOR_ERROR_ADDR(chan) (chan->mmr_base + 0x60) #define XOR_ERROR_ADDR(chan) (chan->mmr_base + 0x60)
#define XOR_INTR_MASK_VALUE 0x3F5 #define XOR_INTR_MASK_VALUE 0x3F5
#define WINDOW_BASE(w) (0x250 + ((w) << 2)) #define WINDOW_BASE(w) (0x50 + ((w) << 2))
#define WINDOW_SIZE(w) (0x270 + ((w) << 2)) #define WINDOW_SIZE(w) (0x70 + ((w) << 2))
#define WINDOW_REMAP_HIGH(w) (0x290 + ((w) << 2)) #define WINDOW_REMAP_HIGH(w) (0x90 + ((w) << 2))
#define WINDOW_BAR_ENABLE(chan) (0x240 + ((chan) << 2)) #define WINDOW_BAR_ENABLE(chan) (0x40 + ((chan) << 2))
#define WINDOW_OVERRIDE_CTRL(chan) (0x2A0 + ((chan) << 2)) #define WINDOW_OVERRIDE_CTRL(chan) (0xA0 + ((chan) << 2))
struct mv_xor_device { struct mv_xor_device {
void __iomem *xor_base; void __iomem *xor_base;
...@@ -82,6 +82,7 @@ struct mv_xor_chan { ...@@ -82,6 +82,7 @@ struct mv_xor_chan {
int pending; int pending;
spinlock_t lock; /* protects the descriptor slot pool */ spinlock_t lock; /* protects the descriptor slot pool */
void __iomem *mmr_base; void __iomem *mmr_base;
void __iomem *mmr_high_base;
unsigned int idx; unsigned int idx;
int irq; int irq;
enum dma_transaction_type current_type; enum dma_transaction_type current_type;
......
This diff is collapsed.
...@@ -248,7 +248,7 @@ static enum dma_status omap_dma_tx_status(struct dma_chan *chan, ...@@ -248,7 +248,7 @@ static enum dma_status omap_dma_tx_status(struct dma_chan *chan,
unsigned long flags; unsigned long flags;
ret = dma_cookie_status(chan, cookie, txstate); ret = dma_cookie_status(chan, cookie, txstate);
if (ret == DMA_SUCCESS || !txstate) if (ret == DMA_COMPLETE || !txstate)
return ret; return ret;
spin_lock_irqsave(&c->vc.lock, flags); spin_lock_irqsave(&c->vc.lock, flags);
......
This diff is collapsed.
This diff is collapsed.
...@@ -436,7 +436,7 @@ static enum dma_status sa11x0_dma_tx_status(struct dma_chan *chan, ...@@ -436,7 +436,7 @@ static enum dma_status sa11x0_dma_tx_status(struct dma_chan *chan,
enum dma_status ret; enum dma_status ret;
ret = dma_cookie_status(&c->vc.chan, cookie, state); ret = dma_cookie_status(&c->vc.chan, cookie, state);
if (ret == DMA_SUCCESS) if (ret == DMA_COMPLETE)
return ret; return ret;
if (!state) if (!state)
......
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment