Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Support
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
E
embulk-input-filename
Project overview
Project overview
Details
Activity
Releases
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Issues
0
Issues
0
List
Boards
Labels
Milestones
Merge Requests
0
Merge Requests
0
Analytics
Analytics
Repository
Value Stream
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Create a new issue
Commits
Issue Boards
Open sidebar
Klaus Wölfel
embulk-input-filename
Commits
de0dfe93
Commit
de0dfe93
authored
Aug 14, 2017
by
yu
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
finished the modification of the sorter
parent
f83e6c97
Changes
6
Expand all
Hide whitespace changes
Inline
Side-by-side
Showing
6 changed files
with
140 additions
and
114 deletions
+140
-114
README.md
README.md
+2
-2
src/main/java/org/embulk/input/filename/FilenameInputPlugin.java
...n/java/org/embulk/input/filename/FilenameInputPlugin.java
+71
-110
src/test/java/org/embulk/input/filename/TestFilenameInputPlugin.java
...va/org/embulk/input/filename/TestFilenameInputPlugin.java
+65
-0
src/test/resources/testModifiedOrder/sample_2.txt
src/test/resources/testModifiedOrder/sample_2.txt
+1
-1
src/test/resources/testModifiedOrder/sample_3.txt
src/test/resources/testModifiedOrder/sample_3.txt
+0
-1
src/test/resources/testModifiedOrder/sample_4.txt
src/test/resources/testModifiedOrder/sample_4.txt
+1
-0
No files found.
README.md
View file @
de0dfe93
...
...
@@ -11,7 +11,7 @@
## Configuration
-
**multi_dir**
: description (ArrayList
<String>
, required)
-
**mul
it
_tag**
: description (ArrayList
<String>
, default:
`[]`
)
-
**mul
ti
_tag**
: description (ArrayList
<String>
, default:
`[]`
)
-
**load_order**
: description (String, default:
`ALPHABETICAL`
)
-
**chunk_size**
: description (int, default:
`10485760(10M)`
)
...
...
@@ -115,7 +115,7 @@ exec:
min_output_tasks
:
1
in
:
type
:
filename
mul
it
_dir
:
[
"
../sample/sample_"
,
"
../example/example_"
]
mul
ti
_dir
:
[
"
../sample/sample_"
,
"
../example/example_"
]
multi_tag
:
[
"
tag1"
,
"
tag2"
]
load_order
:
ASCEND_MODIFIED
chunk_size
:
1000
...
...
src/main/java/org/embulk/input/filename/FilenameInputPlugin.java
View file @
de0dfe93
This diff is collapsed.
Click to expand it.
src/test/java/org/embulk/input/filename/TestFilenameInputPlugin.java
View file @
de0dfe93
...
...
@@ -124,9 +124,14 @@ public class TestFilenameInputPlugin
res
=
embulk
.
runAllBuilder
(
execConfig
,
inConfig
,
outConfig
);
lines
=
Files
.
readAllLines
(
Paths
.
get
(
tmp
.
toString
()+
"/outputfile0.txt"
));
//We reverse the actual files
Collections
.
reverse
(
actual
);
System
.
out
.
println
(
"The lines"
+
lines
);
System
.
out
.
println
(
"The actual"
+
actual
);
assertEquals
(
lines
,
actual
);
}
...
...
@@ -222,6 +227,7 @@ public class TestFilenameInputPlugin
System
.
out
.
println
(
lines1
);
System
.
out
.
println
(
dir1
);
assertEquals
(
lines1
,
dir1
);
assertEquals
(
lines2
,
dir2
);
}
...
...
@@ -298,6 +304,7 @@ public class TestFilenameInputPlugin
System
.
out
.
println
(
lines1
);
System
.
out
.
println
(
dir1
);
assertEquals
(
lines1
,
dir1
);
assertEquals
(
lines2
,
dir2
);
}
...
...
@@ -334,5 +341,63 @@ public class TestFilenameInputPlugin
assertEquals
(
actual
,
lines
);
}
@Test
public
void
testLastPath
()
throws
Exception
{
ConfigSource
execConfig
=
embulk
.
newConfig
()
.
set
(
"min_output_tasks"
,
"1"
);
Path
path_src
=
Paths
.
get
(
"src/test/resources/testDirList"
);
// Be careful the name of the List should be multi_dir!
List
<
String
>
multi_dir
=
Arrays
.
asList
(
path_src
.
toAbsolutePath
().
toString
()+
"/sample/sample_"
,
path_src
.
toAbsolutePath
().
toString
()+
"/example/example_"
);
List
<
String
>
multi_tag
=
Arrays
.
asList
(
"hello"
,
"world"
);
List
<
String
>
lastPaths
=
Arrays
.
asList
(
path_src
.
toAbsolutePath
().
toString
()+
"/sample/sample_02.txt"
,
path_src
.
toAbsolutePath
().
toString
()+
"/example/example_01.txt"
);
ConfigSource
inConfig
=
embulk
.
newConfig
()
.
set
(
"type"
,
"filename"
)
.
set
(
"load_order"
,
"ALPHABETICAL"
)
.
set
(
"lastPaths"
,
lastPaths
)
.
set
(
"multi_dir"
,
multi_dir
);
Path
tmp
=
embulk
.
createTempDir
();
ConfigSource
outConfig
=
embulk
.
newConfig
()
.
set
(
"type"
,
"joinfile"
)
.
set
(
"sum_type"
,
"filename"
)
.
set
(
"path_prefix"
,
tmp
.
toString
()+
"/outputfile"
)
.
set
(
"file_ext"
,
"txt"
);
TestHelper
.
RunResult
res
=
embulk
.
runAllBuilder
(
execConfig
,
inConfig
,
outConfig
);
List
<
String
>
lines1
=
Files
.
readAllLines
(
Paths
.
get
(
tmp
.
toString
()+
"/outputfile0.txt"
));
List
<
String
>
lines2
=
Files
.
readAllLines
(
Paths
.
get
(
tmp
.
toString
()+
"/outputfile1.txt"
));
List
<
String
>
dir1
=
Files
.
walk
(
Paths
.
get
(
path_src
.
toAbsolutePath
().
toString
()+
"/sample"
))
.
filter
(
Files:
:
isRegularFile
)
.
map
(
Path:
:
toAbsolutePath
)
.
map
(
Path:
:
toString
)
.
collect
(
Collectors
.
toList
());
Collections
.
sort
(
dir1
);
List
<
String
>
dir2
=
Files
.
walk
(
Paths
.
get
(
path_src
.
toAbsolutePath
().
toString
()+
"/example"
))
.
filter
(
Files:
:
isRegularFile
)
.
map
(
Path:
:
toAbsolutePath
)
.
map
(
Path:
:
toString
)
.
collect
(
Collectors
.
toList
());
List
<
String
>
fromLastPath
=
dir2
.
subList
(
0
,
dir2
.
size
());
Collections
.
sort
(
dir2
);
//System.out.println(lines1);
//System.out.println(dir1.subList(2,dir1.size()));
//System.out.println(lines2);
//System.out.println(dir2.subList(1,dir2.size()));
assertEquals
(
lines1
,
dir1
.
subList
(
2
,
dir1
.
size
()));
assertEquals
(
lines2
,
dir2
.
subList
(
1
,
dir2
.
size
()));
}
}
src/test/resources/testModifiedOrder/sample_2.txt
View file @
de0dfe93
2
2
2
2
src/test/resources/testModifiedOrder/sample_3.txt
View file @
de0dfe93
...
...
@@ -2,5 +2,4 @@
3
3
3
3
src/test/resources/testModifiedOrder/sample_4.txt
View file @
de0dfe93
4
4
4
4
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment