Skip to content

M
mariadb
Commit d3c44c99 authored by Rich Prohaska's avatar Rich Prohaska
Browse files
Options

DB-708 move test scripts

parent f90c40ef
Expand all Hide whitespace changes
Inline Side-by-side
Showing with 0 additions and 1977 deletions
scripts/atc.ontime/atc_ontime_create.sql deleted 100644 → 0
View file @ f90c40ef
CREATE TABLE `ontime` (
`Year` year(4) DEFAULT NULL,
`Quarter` tinyint(4) DEFAULT NULL,
`Month` tinyint(4) DEFAULT NULL,
`DayofMonth` tinyint(4) DEFAULT NULL,
`DayOfWeek` tinyint(4) DEFAULT NULL,
`FlightDate` date DEFAULT NULL,
`UniqueCarrier` char(7) DEFAULT NULL,
`AirlineID` int(11) DEFAULT NULL,
`Carrier` char(2) DEFAULT NULL,
`TailNum` varchar(50) DEFAULT NULL,
`FlightNum` varchar(10) DEFAULT NULL,
`Origin` char(5) DEFAULT NULL,
`OriginCityName` varchar(100) DEFAULT NULL,
`OriginState` char(2) DEFAULT NULL,
`OriginStateFips` varchar(10) DEFAULT NULL,
`OriginStateName` varchar(100) DEFAULT NULL,
`OriginWac` int(11) DEFAULT NULL,
`Dest` char(5) DEFAULT NULL,
`DestCityName` varchar(100) DEFAULT NULL,
`DestState` char(2) DEFAULT NULL,
`DestStateFips` varchar(10) DEFAULT NULL,
`DestStateName` varchar(100) DEFAULT NULL,
`DestWac` int(11) DEFAULT NULL,
`CRSDepTime` int(11) DEFAULT NULL,
`DepTime` int(11) DEFAULT NULL,
`DepDelay` int(11) DEFAULT NULL,
`DepDelayMinutes` int(11) DEFAULT NULL,
`DepDel15` int(11) DEFAULT NULL,
`DepartureDelayGroups` int(11) DEFAULT NULL,
`DepTimeBlk` varchar(20) DEFAULT NULL,
`TaxiOut` int(11) DEFAULT NULL,
`WheelsOff` int(11) DEFAULT NULL,
`WheelsOn` int(11) DEFAULT NULL,
`TaxiIn` int(11) DEFAULT NULL,
`CRSArrTime` int(11) DEFAULT NULL,
`ArrTime` int(11) DEFAULT NULL,
`ArrDelay` int(11) DEFAULT NULL,
`ArrDelayMinutes` int(11) DEFAULT NULL,
`ArrDel15` int(11) DEFAULT NULL,
`ArrivalDelayGroups` int(11) DEFAULT NULL,
`ArrTimeBlk` varchar(20) DEFAULT NULL,
`Cancelled` tinyint(4) DEFAULT NULL,
`CancellationCode` char(1) DEFAULT NULL,
`Diverted` tinyint(4) DEFAULT NULL,
`CRSElapsedTime` INT(11) DEFAULT NULL,
`ActualElapsedTime` INT(11) DEFAULT NULL,
`AirTime` INT(11) DEFAULT NULL,
`Flights` INT(11) DEFAULT NULL,
`Distance` INT(11) DEFAULT NULL,
`DistanceGroup` TINYINT(4) DEFAULT NULL,
`CarrierDelay` INT(11) DEFAULT NULL,
`WeatherDelay` INT(11) DEFAULT NULL,
`NASDelay` INT(11) DEFAULT NULL,
`SecurityDelay` INT(11) DEFAULT NULL,
`LateAircraftDelay` INT(11) DEFAULT NULL,
`FirstDepTime` varchar(10) DEFAULT NULL,
`TotalAddGTime` varchar(10) DEFAULT NULL,
`LongestAddGTime` varchar(10) DEFAULT NULL,
`DivAirportLandings` varchar(10) DEFAULT NULL,
`DivReachedDest` varchar(10) DEFAULT NULL,
`DivActualElapsedTime` varchar(10) DEFAULT NULL,
`DivArrDelay` varchar(10) DEFAULT NULL,
`DivDistance` varchar(10) DEFAULT NULL,
`Div1Airport` varchar(10) DEFAULT NULL,
`Div1WheelsOn` varchar(10) DEFAULT NULL,
`Div1TotalGTime` varchar(10) DEFAULT NULL,
`Div1LongestGTime` varchar(10) DEFAULT NULL,
`Div1WheelsOff` varchar(10) DEFAULT NULL,
`Div1TailNum` varchar(10) DEFAULT NULL,
`Div2Airport` varchar(10) DEFAULT NULL,
`Div2WheelsOn` varchar(10) DEFAULT NULL,
`Div2TotalGTime` varchar(10) DEFAULT NULL,
`Div2LongestGTime` varchar(10) DEFAULT NULL,
`Div2WheelsOff` varchar(10) DEFAULT NULL,
`Div2TailNum` varchar(10) DEFAULT NULL,
`Div3Airport` varchar(10) DEFAULT NULL,
`Div3WheelsOn` varchar(10) DEFAULT NULL,
`Div3TotalGTime` varchar(10) DEFAULT NULL,
`Div3LongestGTime` varchar(10) DEFAULT NULL,
`Div3WheelsOff` varchar(10) DEFAULT NULL,
`Div3TailNum` varchar(10) DEFAULT NULL,
`Div4Airport` varchar(10) DEFAULT NULL,
`Div4WheelsOn` varchar(10) DEFAULT NULL,
`Div4TotalGTime` varchar(10) DEFAULT NULL,
`Div4LongestGTime` varchar(10) DEFAULT NULL,
`Div4WheelsOff` varchar(10) DEFAULT NULL,
`Div4TailNum` varchar(10) DEFAULT NULL,
`Div5Airport` varchar(10) DEFAULT NULL,
`Div5WheelsOn` varchar(10) DEFAULT NULL,
`Div5TotalGTime` varchar(10) DEFAULT NULL,
`Div5LongestGTime` varchar(10) DEFAULT NULL,
`Div5WheelsOff` varchar(10) DEFAULT NULL,
`Div5TailNum` varchar(10) DEFAULT NULL
) ENGINE=TOKUDB;
\ No newline at end of file
scripts/atc.ontime/atc_ontime_create_covered.sql deleted 100644 → 0
View file @ f90c40ef
CREATE TABLE `ontime` (
`Year` year(4) DEFAULT NULL,
`Quarter` tinyint(4) DEFAULT NULL,
`Month` tinyint(4) DEFAULT NULL,
`DayofMonth` tinyint(4) DEFAULT NULL,
`DayOfWeek` tinyint(4) DEFAULT NULL,
`FlightDate` date DEFAULT NULL,
`UniqueCarrier` char(7) DEFAULT NULL,
`AirlineID` int(11) DEFAULT NULL,
`Carrier` char(2) DEFAULT NULL,
`TailNum` varchar(50) DEFAULT NULL,
`FlightNum` varchar(10) DEFAULT NULL,
`Origin` char(5) DEFAULT NULL,
`OriginCityName` varchar(100) DEFAULT NULL,
`OriginState` char(2) DEFAULT NULL,
`OriginStateFips` varchar(10) DEFAULT NULL,
`OriginStateName` varchar(100) DEFAULT NULL,
`OriginWac` int(11) DEFAULT NULL,
`Dest` char(5) DEFAULT NULL,
`DestCityName` varchar(100) DEFAULT NULL,
`DestState` char(2) DEFAULT NULL,
`DestStateFips` varchar(10) DEFAULT NULL,
`DestStateName` varchar(100) DEFAULT NULL,
`DestWac` int(11) DEFAULT NULL,
`CRSDepTime` int(11) DEFAULT NULL,
`DepTime` int(11) DEFAULT NULL,
`DepDelay` int(11) DEFAULT NULL,
`DepDelayMinutes` int(11) DEFAULT NULL,
`DepDel15` int(11) DEFAULT NULL,
`DepartureDelayGroups` int(11) DEFAULT NULL,
`DepTimeBlk` varchar(20) DEFAULT NULL,
`TaxiOut` int(11) DEFAULT NULL,
`WheelsOff` int(11) DEFAULT NULL,
`WheelsOn` int(11) DEFAULT NULL,
`TaxiIn` int(11) DEFAULT NULL,
`CRSArrTime` int(11) DEFAULT NULL,
`ArrTime` int(11) DEFAULT NULL,
`ArrDelay` int(11) DEFAULT NULL,
`ArrDelayMinutes` int(11) DEFAULT NULL,
`ArrDel15` int(11) DEFAULT NULL,
`ArrivalDelayGroups` int(11) DEFAULT NULL,
`ArrTimeBlk` varchar(20) DEFAULT NULL,
`Cancelled` tinyint(4) DEFAULT NULL,
`CancellationCode` char(1) DEFAULT NULL,
`Diverted` tinyint(4) DEFAULT NULL,
`CRSElapsedTime` INT(11) DEFAULT NULL,
`ActualElapsedTime` INT(11) DEFAULT NULL,
`AirTime` INT(11) DEFAULT NULL,
`Flights` INT(11) DEFAULT NULL,
`Distance` INT(11) DEFAULT NULL,
`DistanceGroup` TINYINT(4) DEFAULT NULL,
`CarrierDelay` INT(11) DEFAULT NULL,
`WeatherDelay` INT(11) DEFAULT NULL,
`NASDelay` INT(11) DEFAULT NULL,
`SecurityDelay` INT(11) DEFAULT NULL,
`LateAircraftDelay` INT(11) DEFAULT NULL,
`FirstDepTime` varchar(10) DEFAULT NULL,
`TotalAddGTime` varchar(10) DEFAULT NULL,
`LongestAddGTime` varchar(10) DEFAULT NULL,
`DivAirportLandings` varchar(10) DEFAULT NULL,
`DivReachedDest` varchar(10) DEFAULT NULL,
`DivActualElapsedTime` varchar(10) DEFAULT NULL,
`DivArrDelay` varchar(10) DEFAULT NULL,
`DivDistance` varchar(10) DEFAULT NULL,
`Div1Airport` varchar(10) DEFAULT NULL,
`Div1WheelsOn` varchar(10) DEFAULT NULL,
`Div1TotalGTime` varchar(10) DEFAULT NULL,
`Div1LongestGTime` varchar(10) DEFAULT NULL,
`Div1WheelsOff` varchar(10) DEFAULT NULL,
`Div1TailNum` varchar(10) DEFAULT NULL,
`Div2Airport` varchar(10) DEFAULT NULL,
`Div2WheelsOn` varchar(10) DEFAULT NULL,
`Div2TotalGTime` varchar(10) DEFAULT NULL,
`Div2LongestGTime` varchar(10) DEFAULT NULL,
`Div2WheelsOff` varchar(10) DEFAULT NULL,
`Div2TailNum` varchar(10) DEFAULT NULL,
`Div3Airport` varchar(10) DEFAULT NULL,
`Div3WheelsOn` varchar(10) DEFAULT NULL,
`Div3TotalGTime` varchar(10) DEFAULT NULL,
`Div3LongestGTime` varchar(10) DEFAULT NULL,
`Div3WheelsOff` varchar(10) DEFAULT NULL,
`Div3TailNum` varchar(10) DEFAULT NULL,
`Div4Airport` varchar(10) DEFAULT NULL,
`Div4WheelsOn` varchar(10) DEFAULT NULL,
`Div4TotalGTime` varchar(10) DEFAULT NULL,
`Div4LongestGTime` varchar(10) DEFAULT NULL,
`Div4WheelsOff` varchar(10) DEFAULT NULL,
`Div4TailNum` varchar(10) DEFAULT NULL,
`Div5Airport` varchar(10) DEFAULT NULL,
`Div5WheelsOn` varchar(10) DEFAULT NULL,
`Div5TotalGTime` varchar(10) DEFAULT NULL,
`Div5LongestGTime` varchar(10) DEFAULT NULL,
`Div5WheelsOff` varchar(10) DEFAULT NULL,
`Div5TailNum` varchar(10) DEFAULT NULL,
KEY `Year` (`Year`,`Month`),
KEY `Year_2` (`Year`,`DayOfWeek`),
KEY `Year_3` (`Year`,`DepDelay`,`DayOfWeek`),
KEY `DayOfWeek` (`DayOfWeek`,`Year`,`DepDelay`),
KEY `Year_4` (`Year`,`DepDelay`,`Origin`,`Carrier`),
KEY `DepDelay` (`DepDelay`,`Year`),
KEY `Year_5` (`Year`,`DestCityName`,`OriginCityName`),
KEY `DestCityName` (`DestCityName`,`OriginCityName`,`Year`)
) ENGINE=TOKUDB;
\ No newline at end of file
scripts/atc.ontime/nodistinct.q8.sql deleted 100644 → 0
View file @ f90c40ef
# Q8: As final I tested most popular destination in sense count of direct connected cities for different diapason of years.
SELECT DestCityName, COUNT( OriginCityName) FROM ontime WHERE Year BETWEEN 2006 and 2007 GROUP BY DestCityName ORDER BY 2 DESC LIMIT 10;
scripts/atc.ontime/q0.sql deleted 100644 → 0
View file @ f90c40ef
select avg(c1) from (select year,month,count(*) as c1 from ontime group by YEAR,month) t;
scripts/atc.ontime/q1.result deleted 100644 → 0
View file @ f90c40ef
DayOfWeek c
5 8732424
1 8730614
4 8710843
3 8685626
2 8639632
7 8274367
6 7514194
scripts/atc.ontime/q1.sql deleted 100644 → 0
View file @ f90c40ef
SELECT DayOfWeek, count(*) AS c FROM ontime WHERE Year BETWEEN 2000 AND 2008 GROUP BY DayOfWeek ORDER BY c DESC;
scripts/atc.ontime/q2.result deleted 100644 → 0
View file @ f90c40ef
DayOfWeek c
5 2088300
4 1918325
1 1795120
7 1782292
3 1640798
2 1538291
6 1391984
scripts/atc.ontime/q2.sql deleted 100644 → 0
View file @ f90c40ef
SELECT DayOfWeek, count(*) AS c FROM ontime WHERE DepDelay>10 AND Year BETWEEN 2000 AND 2008 GROUP BY DayOfWeek ORDER BY c DESC;
scripts/atc.ontime/q3.result deleted 100644 → 0
View file @ f90c40ef
Origin c
ORD 846692
ATL 822955
DFW 601318
LAX 391247
PHX 391191
LAS 351713
DEN 345108
EWR 292916
DTW 289233
IAH 283861
scripts/atc.ontime/q3.sql deleted 100644 → 0
View file @ f90c40ef
SELECT Origin, count(*) AS c FROM ontime WHERE DepDelay>10 AND Year BETWEEN 2000 AND 2008 GROUP BY Origin ORDER BY c DESC LIMIT 10;
scripts/atc.ontime/q4.result deleted 100644 → 0
View file @ f90c40ef
carrier count(*)
WN 296293
AA 176203
MQ 145630
US 135987
UA 128174
OO 127426
EV 101796
XE 99915
DL 93675
NW 90429
CO 76662
YV 67905
FL 59460
OH 59034
B6 50740
9E 46948
AS 42830
F9 23035
AQ 4299
HA 2746
scripts/atc.ontime/q4.sql deleted 100644 → 0
View file @ f90c40ef
SELECT carrier, count(*) FROM ontime WHERE DepDelay>10 AND Year=2007 GROUP BY carrier ORDER BY 2 DESC;
scripts/atc.ontime/q5.result deleted 100644 → 0
View file @ f90c40ef
carrier c c2 c3
EV 101796 286234 355.6391
US 135987 485447 280.1274
AA 176203 633857 277.9854
MQ 145630 540494 269.4387
AS 42830 160185 267.3783
B6 50740 191450 265.0300
UA 128174 490002 261.5785
WN 296293 1168871 253.4865
OH 59034 236032 250.1102
CO 76662 323151 237.2327
F9 23035 97760 235.6281
YV 67905 294362 230.6853
XE 99915 434773 229.8096
FL 59460 263159 225.9471
NW 90429 414526 218.1504
OO 127426 597880 213.1297
DL 93675 475889 196.8421
9E 46948 258851 181.3707
AQ 4299 46360 92.7308
HA 2746 56175 48.8830
scripts/atc.ontime/q5.sql deleted 100644 → 0
View file @ f90c40ef
SELECT t.carrier, c, c2, c*1000/c2 as c3 FROM (SELECT carrier, count(*) AS c FROM ontime WHERE DepDelay>10 AND Year=2007 GROUP BY carrier) t JOIN (SELECT carrier, count(*) AS c2 FROM ontime WHERE Year=2007 GROUP BY carrier) t2 ON (t.Carrier=t2.Carrier) ORDER BY c3 DESC;
scripts/atc.ontime/q6.result deleted 100644 → 0
View file @ f90c40ef
carrier c c2 c3
UA 1096646 490002 2238.0439
AS 354145 160185 2210.8500
DL 1050448 475889 2207.3383
AA 1276555 633857 2013.9479
US 909154 485447 1872.8182
WN 2165483 1168871 1852.6279
NW 725076 414526 1749.1689
MQ 876799 540494 1622.2178
CO 522219 323151 1616.0216
EV 461050 286234 1610.7451
OH 301681 236032 1278.1360
FL 298916 263159 1135.8760
B6 197249 191450 1030.2899
OO 556247 597880 930.3656
F9 72150 97760 738.0319
YV 198787 294362 675.3147
XE 233488 434773 537.0343
AQ 17239 46360 371.8507
9E 89391 258851 345.3377
HA 15968 56175 284.2546
scripts/atc.ontime/q6.sql deleted 100644 → 0
View file @ f90c40ef
SELECT t.carrier, c, c2, c*1000/c2 as c3 FROM (SELECT carrier, count(*) AS c FROM ontime WHERE DepDelay>10 AND Year BETWEEN 2000 and 2008 GROUP BY carrier) t JOIN (SELECT carrier, count(*) AS c2 FROM ontime WHERE Year=2007 GROUP BY carrier) t2 ON (t.Carrier=t2.Carrier) ORDER BY c3 DESC;
scripts/atc.ontime/q7.result deleted 100644 → 0
View file @ f90c40ef
Year c1/c2
1988 166.1709
1989 199.5009
1990 166.4513
1991 147.2163
1992 146.7543
1993 154.2498
1994 165.6803
1995 193.9344
1996 221.8281
1997 191.6513
1998 193.5638
1999 200.8742
2000 231.7167
2001 189.0581
2002 162.3769
2003 150.2455
2004 192.4838
2005 207.5929
2006 231.5599
2007 245.3487
2008 219.9228
scripts/atc.ontime/q7.sql deleted 100644 → 0
View file @ f90c40ef
SELECT t.Year, c1/c2 FROM (select Year,count(*)*1000 as c1 from ontime WHERE DepDelay>10 GROUP BY Year) t JOIN (select Year,count(*) as c2 from ontime GROUP BY Year) t2 ON (t.Year=t2.Year);
scripts/atc.ontime/q8.10y.destcityname.result deleted 100644 → 0
View file @ f90c40ef
DestCityName COUNT( DISTINCT OriginCityName)
Atlanta, GA 190
Chicago, IL 159
Dallas/Ft.Worth, TX 151
Cincinnati, OH 139
Minneapolis, MN 131
Houston, TX 127
Detroit, MI 121
Denver, CO 120
Salt Lake City, UT 116
New York, NY 111
scripts/atc.ontime/q8.10y.destcityname.sql deleted 100644 → 0
View file @ f90c40ef
SELECT DestCityName, COUNT( DISTINCT OriginCityName) FROM ontime FORCE INDEX(DestCityName) WHERE Year BETWEEN 1999 and 2009 GROUP BY DestCityName ORDER BY 2 DESC LIMIT 10;
scripts/atc.ontime/q8.10y.result deleted 100644 → 0
View file @ f90c40ef
DestCityName COUNT( DISTINCT OriginCityName)
Atlanta, GA 190
Chicago, IL 159
Dallas/Ft.Worth, TX 151
Cincinnati, OH 139
Minneapolis, MN 131
Houston, TX 127
Detroit, MI 121
Denver, CO 120
Salt Lake City, UT 116
New York, NY 111
scripts/atc.ontime/q8.10y.sql deleted 100644 → 0
View file @ f90c40ef
SELECT DestCityName, COUNT( DISTINCT OriginCityName) FROM ontime WHERE Year BETWEEN 1999 and 2009 GROUP BY DestCityName ORDER BY 2 DESC LIMIT 10;
scripts/atc.ontime/q8.1y.sql deleted 100644 → 0
View file @ f90c40ef
SELECT DestCityName, COUNT( DISTINCT OriginCityName) FROM ontime WHERE Year BETWEEN 1999 and 1999 GROUP BY DestCityName ORDER BY 2 DESC LIMIT 10;
scripts/atc.ontime/q8.1y.year5.sql deleted 100644 → 0
View file @ f90c40ef
SELECT DestCityName, COUNT( DISTINCT OriginCityName) FROM ontime USE INDEX(year_5) WHERE Year BETWEEN 1999 and 1999 GROUP BY DestCityName ORDER BY 2 DESC LIMIT 10;
scripts/atc.ontime/q8.2y.sql deleted 100644 → 0
View file @ f90c40ef
SELECT DestCityName, COUNT( DISTINCT OriginCityName) FROM ontime WHERE Year BETWEEN 1999 and 2000 GROUP BY DestCityName ORDER BY 2 DESC LIMIT 10;
scripts/atc.ontime/q8.3y.sql deleted 100644 → 0
View file @ f90c40ef
SELECT DestCityName, COUNT( DISTINCT OriginCityName) FROM ontime WHERE Year BETWEEN 1999 and 2001 GROUP BY DestCityName ORDER BY 2 DESC LIMIT 10;
scripts/atc.ontime/q8.4y.sql deleted 100644 → 0
View file @ f90c40ef
SELECT DestCityName, COUNT( DISTINCT OriginCityName) FROM ontime WHERE Year BETWEEN 1999 and 2002 GROUP BY DestCityName ORDER BY 2 DESC LIMIT 10;
scripts/atc.ontime/q8.result deleted 100644 → 0
View file @ f90c40ef
DestCityName COUNT( DISTINCT OriginCityName)
Atlanta, GA 183
Chicago, IL 147
Dallas/Ft.Worth, TX 133
Cincinnati, OH 129
Minneapolis, MN 128
Houston, TX 114
Detroit, MI 112
Denver, CO 111
Salt Lake City, UT 108
New York, NY 101
scripts/atc.ontime/q8.sql deleted 100644 → 0
View file @ f90c40ef
SELECT DestCityName, COUNT( DISTINCT OriginCityName) FROM ontime WHERE Year BETWEEN 2006 and 2007 GROUP BY DestCityName ORDER BY 2 DESC LIMIT 10;
scripts/atc.ontime/q8.year5.sql deleted 100644 → 0
View file @ f90c40ef
SELECT DestCityName, COUNT( DISTINCT OriginCityName) FROM ontime use index(year_5) WHERE Year BETWEEN 2006 and 2007 GROUP BY DestCityName ORDER BY 2 DESC LIMIT 10;
scripts/atc.ontime/q9.result deleted 100644 → 0
View file @ f90c40ef
year c1
1988 5202096
1989 5041200
1990 5270893
1991 5076925
1992 5092157
1993 5070501
1994 5180048
1995 5327435
1996 5351983
1997 5411843
1998 5384721
1999 5527884
2000 5683047
2001 5967780
2002 5271359
2003 6488540
2004 7129270
2005 7140596
2006 7141922
2007 7455458
2008 7009728
scripts/atc.ontime/q9.sql deleted 100644 → 0
View file @ f90c40ef
select year,count(*) as c1 from ontime group by YEAR;
scripts/atc.ontime/qcount.main.sql deleted 100644 → 0
View file @ f90c40ef
select count(*) from ontime use index();
scripts/atc.ontime/qcount.sql deleted 100644 → 0
View file @ f90c40ef
select count(*) from ontime;
scripts/atc.readme deleted 100644 → 0
View file @ f90c40ef
The script to run the load the air traffic ontime database and run queries against it
is called run.atc.ontime.bas.
The queries are in the tokudb-engine/scripts/atc.ontime directory.
The data for the ontime database is in the amazon s3 bucket called tokutek-mysql-data.
$ s3ls -l tokutek-mysql-data --prefix=atc_On_Time_Performance
2010-06-15T13:07:09.000Z 1073741824 atc_On_Time_Performance.mysql.csv.gz.aa
2010-06-15T13:08:19.000Z 1073741824 atc_On_Time_Performance.mysql.csv.gz.ab
2010-06-15T13:09:38.000Z 1073741824 atc_On_Time_Performance.mysql.csv.gz.ac
2010-06-15T13:10:54.000Z 446709742 atc_On_Time_Performance.mysql.csv.gz.ad
2010-06-15T13:11:26.000Z 503 atc_On_Time_Performance.mysql.csv.gz.xml
The raw data is also stored in the amazon s3 bucket called tokutek-mysql-data.
$ s3ls -l tokutek-mysql-data --prefix=atc
scripts/nightly.mysql.build.and.test.bash deleted 100755 → 0
View file @ f90c40ef
#!/usr/bin/env bash
function usage() {
echo "run nightly mysql and fractal tree regressions"
echo "uses gearman to schedule jobs onto test machines"
}
# generate a script that makes a mysql release and run tests on it
function make_and_test_mysql() {
echo $(date) $* >>$nightlytrace 2>&1
echo "bash -x \$HOME/github/tokudb-engine/scripts/tokutek.make.mysql.bash $* >>$mysqltrace 2>&1; \
buildexitcode=\$?; \
echo \$(date) \$HOME/github/tokudb-engine/scripts/tokutek.make.mysql.bash -$* \$buildexitcode >>$mysqltrace; \
if [ \$buildexitcode -eq 0 ] ; then \$HOME/bin/test.mysql.bash $* >>/tmp/mysql.test.trace 2>&1; fi" \
| $gearmandir/bin/gearman -b -f mysql-build-$system-$arch -h $gearmandhost -p 4730 >>$nightlytrace 2>&1
}
# make a mysql release
function make_mysql() {
echo $(date) $* >>$nightlytrace 2>&1
echo "\$HOME/github/tokudb-engine/scripts/tokutek.make.mysql.bash $* >>$mysqltrace 2>&1" | $gearmandir/bin/gearman -b -f mysql-build-$system-$arch -h $gearmandhost -p 4730 >>$nightlytrace 2>&1
}
# setup the PATH since cron gives us a minimal PATH
PATH=$HOME/bin:$HOME/usr/local/bin:/usr/local/bin:$PATH
source /etc/profile
github_token=
gearmandhost=localhost
gearmandir=/usr/local/gearmand-1.1.6
system=$(uname -s | tr '[:upper:]' '[:lower:]')
arch=$(uname -m | tr '[:upper:]' '[:lower:]')
now_ts=$(date +%s)
cc=gcc
cxx=g++
while [ $# -gt 0 ] ; do
arg=$1; shift
if [[ $arg =~ --(.*)=(.*) ]] ; then
eval ${BASH_REMATCH[1]}=${BASH_REMATCH[2]}
else
usage; exit 1;
fi
done
nightlytrace=/tmp/$(whoami).nightly.trace
mysqltrace=/tmp/$(whoami).mysql.build.trace.$now_ts
make_and_test_mysql --mysqlbuild=mysql-5.6.16-tokudb-${now_ts}-debug-e-${system}-${arch} --cc=$cc --cxx=$cxx --github_token=$github_token
make_and_test_mysql --mysqlbuild=mysql-5.6.16-tokudb-${now_ts}-e-${system}-${arch} --cc=$cc --cxx=$cxx --github_token=$github_token --tests=run.mysql.tests.bash:run.sql.bench.bash
make_and_test_mysql --mysqlbuild=mysql-5.5.36-tokudb-${now_ts}-debug-e-${system}-${arch} --cc=$cc --cxx=$cxx --github_token=$github_token
make_and_test_mysql --mysqlbuild=mysql-5.5.36-tokudb-${now_ts}-e-${system}-${arch} --cc=$cc --cxx=$cxx --github_token=$github_token --tests=run.mysql.tests.bash:run.sql.bench.bash
make_and_test_mysql --mysqlbuild=mariadb-5.5.35-tokudb-${now_ts}-debug-e-${system}-${arch} --cc=$cc --cxx=$cxx --github_token=$github_token
make_and_test_mysql --mysqlbuild=mariadb-5.5.35-tokudb-${now_ts}-e-${system}-${arch} --cc=$cc --cxx=$cxx --github_token=$github_token --tests=run.mysql.tests.bash:run.sql.bench.bash
exit 0
scripts/nightly.mysql.build.and.test.my.cnf deleted 100644 → 0
View file @ f90c40ef
[mysqld]
tmpdir=/data/mysql/tmp
max_connections=1024
table_open_cache=1024
loose_tokudb_cache_size=8G
loose_tokudb_directio=1
scripts/run.atc.ontime.bash deleted 100755 → 0
View file @ f90c40ef
#!/usr/bin/env bash
function usage() {
echo "run the atc ontime load and run"
echo "--mysqlbuild=$mysqlbuild"
echo "[--commit=$commit]"
echo "[--dbname=$dbname]"
echo "[--load=$load] [--check=$check] [--run=$run]"
echo "[--engine=$engine]"
echo "[--tokudb_load_save_space=$tokudb_load_save_space] [--tokudb_row_format=$tokudb_row_format] [--tokudb_loader_memory_size=$tokudb_loader_memory_size]"
}
function retry() {
local cmd
local retries
local exitcode
cmd=$*
let retries=0
while [ $retries -le 10 ] ; do
echo `date` $cmd
bash -c "$cmd"
exitcode=$?
echo `date` $cmd $exitcode $retries
let retries=retries+1
if [ $exitcode -eq 0 ] ; then break; fi
sleep 10
done
test $exitcode = 0
}
mysqlbuild=
commit=0
mysqlserver=`hostname`
mysqluser=`whoami`
mysqlsocket=/tmp/mysql.sock
svnserver=https://svn.tokutek.com/tokudb
basedir=$HOME/svn.build
builddir=$basedir/mysql.build
dbname=atc
tblname=ontime
load=1
check=1
run=1
engine=tokudb
tokudb_load_save_space=0
tokudb_row_format=
tokudb_loader_memory_size=
verbose=0
svn_server=https://svn.tokutek.com/tokudb
svn_branch=.
svn_revision=HEAD
# parse the command line
while [ $# -gt 0 ] ; do
arg=$1; shift
if [[ $arg =~ --(.*)=(.*) ]] ; then
eval ${BASH_REMATCH[1]}=${BASH_REMATCH[2]}
else
usage; exit 1
fi
done
if [[ $mysqlbuild =~ (.*)-(tokudb\-.*)-(linux)-(x86_64) ]] ; then
mysql=${BASH_REMATCH[1]}
tokudb=${BASH_REMATCH[2]}
system=${BASH_REMATCH[3]}
arch=${BASH_REMATCH[4]}
else
exit 1
fi
if [ -d /usr/local/mysql/bin ] ; then
export PATH=/usr/local/mysql/bin:$PATH
fi
if [ -d /usr/local/mysql/lib/mysql ] ; then
export LD_LIBRARY_PATH=/usr/local/mysql/lib/mysql:$PATH
fi
# goto the base directory
if [ ! -d $basedir ] ; then mkdir $basedir; fi
pushd $basedir
# update the build directory
if [ ! -d $builddir ] ; then mkdir $builddir; fi
date=`date +%Y%m%d`
testresultsdir=$builddir/$date
pushd $builddir
while [ ! -d $date ] ; do
svn mkdir $svn_server/mysql.build/$date -m ""
svn checkout $svn_server/mysql.build/$date
if [ $? -ne 0 ] ; then rm -rf $date; fi
done
popd
if [ $dbname = "atc" -a $engine != "tokudb" ] ; then dbname="atc_$engine"; fi
runfile=$testresultsdir/$dbname-$tblname-$mysqlbuild-$mysqlserver
if [ $tokudb_load_save_space != 0 ] ; then runfile=$runfile-compress; fi
if [ "$tokudb_row_format" != "" ] ; then runfile=$runfile-$tokudb_row_format; fi
if [ "$tokudb_loader_memory_size" != "" ] ; then runfile=$runfile-$tokudb_loader_memory_size; fi
rm -rf $runfile
testresult="PASS"
# maybe get the atc data from s3
if [ $testresult = "PASS" ] ; then
f=atc_On_Time_Performance.mysql.csv
if [ ! -f $f ] ; then
f=$f.gz
if [ ! -f $f ] ; then
echo `date` s3get --bundle tokutek-mysql-data $f >>$runfile 2>&1
s3get --verbose --bundle tokutek-mysql-data $f >>$runfile 2>&1
exitcode=$?
echo `date` s3get --bundle tokutek-mysql-data $f $exitcode >>$runfile 2>&1
if [ $exitcode -ne 0 ] ; then testresult="FAIL"; fi
if [ $testresult = "PASS" ] ; then
echo `date` gunzip $f >>$runfile 2>&1
gunzip $f
exitcode=$?
echo `date` gunzip $f $exitcode >>$runfile 2>&1
if [ $exitcode -ne 0 ] ; then testresult="FAIL"; fi
fi
fi
fi
fi
# checkout the atc test from svn
atc=atc-$mysqlbuild
if [ $testresult = "PASS" ] ; then
if [ -d atc-$mysqlbuild ] ; then rm -rf atc-$mysqlbuild; fi
retry svn export -r $svn_revision $svn_server/$svn_branch/mysql/tests/atc atc-$mysqlbuild
exitcode=$?
echo `date` svn export -r $svn_revision $svn_server/$svn_branch/mysql/tests/atc $exitcode >>$runfile 2>&1
if [ $exitcode != 0 ] ; then
retry svn export -r $svn_revision $svn_server/mysql/tests/atc atc-$mysqlbuild
exitcode=$?
echo `date` svn export -r $svn_revision $svn_server/mysql/tests/atc $exitcode >>$runfile 2>&1
fi
if [ $exitcode != 0 ] ; then testresult="FAIL"; fi
fi
# create the database
if [ $load -ne 0 -a $testresult = "PASS" ] ; then
echo `date` drop database if exists $dbname >>$runfile
mysql -S $mysqlsocket -u $mysqluser -e "drop database if exists $dbname" >>$runfile 2>&1
exitcode=$?
echo `date` drop database if exists $dbname $exitcode>>$runfile
if [ $exitcode -ne 0 ] ; then testresult="FAIL"; fi
echo `date` create database $dbname >>$runfile
mysql -S $mysqlsocket -u $mysqluser -e "create database $dbname" >>$runfile 2>&1
exitcode=$?
echo `date` create database $dbname $exitcode >>$runfile
if [ $exitcode -ne 0 ] ; then testresult="FAIL"; fi
fi
# create the table
if [ $load -ne 0 -a $testresult = "PASS" ] ; then
echo `date` create table $dbname.$tblname >>$runfile
mysql -S $mysqlsocket -u $mysqluser -D $dbname -e "source $atc/atc_ontime_create_covered.sql" >>$runfile 2>&1
exitcode=$?
echo `date` create table $exitcode >>$runfile
if [ $exitcode -ne 0 ] ; then testresult="FAIL"; fi
fi
if [ $load -ne 0 -a $testresult = "PASS" -a "$tokudb_row_format" != "" ] ; then
echo `date` create table $dbname.$tblname >>$runfile
mysql -S $mysqlsocket -u $mysqluser -D $dbname -e "alter table $tblname row_format=$tokudb_row_format" >>$runfile 2>&1
exitcode=$?
echo `date` create table $exitcode >>$runfile
if [ $exitcode -ne 0 ] ; then testresult="FAIL"; fi
fi
if [ $load -ne 0 -a $testresult = "PASS" -a $engine != "tokudb" ] ; then
echo `date` alter table $engine >>$runfile
mysql -S $mysqlsocket -u $mysqluser -D $dbname -e "alter table $tblname engine=$engine" >>$runfile 2>&1
exitcode=$?
echo `date` alter table $engine $exitcode >>$runfile
if [ $exitcode -ne 0 ] ; then testresult="FAIL"; fi
fi
if [ $testresult = "PASS" ] ; then
mysql -S $mysqlsocket -u $mysqluser -D $dbname -e "show create table $tblname" >>$runfile 2>&1
fi
if [ $testresult = "PASS" ] ; then
let default_loader_memory_size="$(mysql -S $mysqlsocket -u $mysqluser -e'select @@tokudb_loader_memory_size' --silent --skip-column-names)"
exitcode=$?
echo `date` get tokudb_loader_memory_size $exitcode >>$runfile
if [ $exitcode -ne 0 ] ; then testresult="FAIL"; fi
if [ "$tokudb_loader_memory_size" = "" ] ; then tokudb_loader_memory_size=$default_loader_memory_size; fi
fi
# load the data
if [ $load -ne 0 -a $testresult = "PASS" ] ; then
echo `date` load data >>$runfile
start=$(date +%s)
mysql -S $mysqlsocket -u $mysqluser -D $dbname -e "set tokudb_loader_memory_size=$tokudb_loader_memory_size;\
set tokudb_load_save_space=$tokudb_load_save_space; load data infile '$basedir/atc_On_Time_Performance.mysql.csv' into table $tblname" >>$runfile 2>&1
exitcode=$?
let loadtime=$(date +%s)-$start
echo `date` load data loadtime=$loadtime $exitcode >>$runfile
if [ $exitcode -ne 0 ] ; then testresult="FAIL"; fi
fi
# check the tables
if [ $check -ne 0 -a $testresult = "PASS" ] ; then
echo `date` check table $tblname >> $runfile
mysql -S $mysqlsocket -u $mysqluser -D $dbname -e "check table $tblname" >>$runfile 2>&1
exitcode=$?
echo `date` check table $tblname $exitcode >> $runfile
if [ $exitcode -ne 0 ] ; then testresult="FAIL"; fi
fi
# run the queries
if [ $run -ne 0 -a $testresult = "PASS" ] ; then
pushd $atc
for qfile in q*.sql ; do
if [[ $qfile =~ q(.*)\.sql ]] ; then
qname=${BASH_REMATCH[1]}
q=`cat $qfile`
qrun=q${qname}.run
echo `date` explain $qfile >>$runfile
if [ $verbose -ne 0 ] ; then echo explain $q >>$runfile; fi
mysql -S $mysqlsocket -u $mysqluser -D $dbname -e "explain $q" >$qrun
exitcode=$?
echo `date` explain $qfile $exitcode >>$runfile
if [ $verbose -ne 0 ] ; then cat $qrun >>$runfile; fi
echo `date` $qfile >>$runfile
start=$(date +%s)
if [ $verbose -ne 0 ] ; then echo $q >>$runfile; fi
mysql -S $mysqlsocket -u $mysqluser -D $dbname -e "$q" >$qrun
exitcode=$?
let qtime=$(date +%s)-$start
echo `date` $qfile qtime=$qtime $exitcode >>$runfile
if [ $verbose -ne 0 ] ; then cat $qrun >>$runfile; fi
if [ $exitcode -ne 0 ] ; then
testresult="FAIL"
else
if [ -f q${qname}.result ] ; then
diff $qrun q${qname}.result >>$runfile
exitcode=$?
if [ $exitcode -ne 0 ] ; then
testresult="FAIL"
fi
fi
fi
fi
done
popd
fi
# commit results
if [ $commit != 0 ] ; then
svn add $runfile
retry svn commit -m \"$testresult $dbname $tblname $mysqlbuild $mysqlserver\" $runfile
fi
popd
if [ $testresult = "PASS" ] ; then exitcode=0; else exitcode=1; fi
exit $exitcode
scripts/run.iibench.bash deleted 100755 → 0
View file @ f90c40ef
#!/usr/bin/env bash
function usage() {
echo "run iibench"
echo "--mysqlbuild=$mysqlbuild"
echo "[--max_row=$max_rows] [--rows_per_report=$rows_per_report] [--insert_only=$insert_only] [ --check=$check]"
echo "[--commit=$commit]"
}
function retry() {
local cmd=$*
local retries
local exitcode
let retries=0
while [ $retries -le 10 ] ; do
echo `date` $cmd
bash -c "$cmd"
exitcode=$?
echo `date` $cmd $exitcode $retries
let retries=retries+1
if [ $exitcode -eq 0 ] ; then break; fi
sleep 10
done
test $exitcode = 0
}
mysqlbuild=
commit=0
check=1
mysqlserver=`hostname`
mysqluser=`whoami`
mysqlsocket=/tmp/mysql.sock
svn_server=https://svn.tokutek.com/tokudb
svn_branch=.
svn_revision=HEAD
basedir=$HOME/svn.build
builddir=$basedir/mysql.build
system=`uname -s | tr [:upper:] [:lower:]`
instancetype=
testinstance=
arch=`uname -m | tr [:upper:] [:lower:]`
tracefile=/tmp/run.iibench.trace
cmd=iibench
dbname=$cmd
engine=tokudb
tblname=testit
max_rows=50000000
rows_per_report=1000000
insert_only=1
# parse the command line
while [ $# -gt 0 ] ; do
arg=$1; shift
if [ $arg = "--replace_into" ] ; then
cmd=replace_into
elif [ $arg = "--insert_ignore" ] ; then
cmd=insert_ignore
elif [[ $arg =~ --(.*)=(.*) ]] ; then
eval ${BASH_REMATCH[1]}=${BASH_REMATCH[2]}
else
usage; exit 1
fi
done
# setup the dbname
if [ $dbname = "iibench" ] ; then dbname=${cmd}_${engine}; fi
if [ "$testinstance" != "" ] ; then dbname=${dbname}_${testinstance}; fi
if [ -d /usr/local/mysql ] ; then
export PATH=/usr/local/mysql/bin:$PATH
fi
if [ -d /usr/local/mysql/lib/mysql ] ; then
export LD_LIBRARY_PATH=/usr/local/mysql/lib/mysql:$PATH
fi
# goto the base directory
if [ ! -d $basedir ] ; then mkdir $basedir; fi
pushd $basedir
# update the build directory
if [ $commit != 0 ] ; then
if [ ! -d $builddir ] ; then mkdir $builddir; fi
date=`date +%Y%m%d`
testresultsdir=$builddir/$date
pushd $builddir
while [ ! -d $date ] ; do
svn mkdir $svn_server/mysql.build/$date -m ""
svn checkout -q $svn_server/mysql.build/$date
if [ $? -ne 0 ] ; then rm -rf $date; fi
done
popd
else
testresultsdir=$PWD
fi
# checkout the code
testdir=iibench-$mysqlbuild-$mysqlserver
if [ "$testinstance" != "" ] ; then testdir=$testdir-$testinstance; fi
rm -rf $testdir
retry svn export -q -r $svn_revision $svn_server/$svn_branch/iibench $testdir
exitcode=$?
if [ $exitcode != 0 ] ; then
retry svn export -q -r $svn_revision $svn_server/iibench $testdir
exitcode=$?
fi
if [ $exitcode != 0 ] ; then exit 1; fi
# create the iibench database
mysql -S $mysqlsocket -u root -e "grant all on *.* to '$mysqluser'@'$mysqlserver'"
exitcode=$?
if [ $exitcode != 0 ] ; then exit 1; fi
mysql -S $mysqlsocket -u $mysqluser -e "drop database if exists $dbname"
exitcode=$?
if [ $exitcode != 0 ] ; then exit 1; fi
mysql -S $mysqlsocket -u $mysqluser -e "create database $dbname"
exitcode=$?
if [ $exitcode != 0 ] ; then exit 1; fi
# run
if [ $cmd = "iibench" -a $insert_only != 0 ] ; then
runfile=$testresultsdir/$dbname-insert_only-$max_rows-$mysqlbuild-$mysqlserver
else
runfile=$testresultsdir/$dbname-$max_rows-$mysqlbuild-$mysqlserver
fi
if [ "$instancetype" != "" ] ; then runfile=$runfile-$instancetype; fi
testresult="PASS"
pushd $testdir/py
echo `date` $cmd start $mysqlbuild $svn_branch $svn_revision $max_rows $rows_per_report >>$runfile
runcmd=$cmd.py
args="--db_user=$mysqluser --db_name=$dbname --db_socket=$mysqlsocket --engine=$engine --setup --max_rows=$max_rows --rows_per_report=$rows_per_report --table_name=$tblname"
if [ $cmd = "iibench" -a $insert_only != 0 ] ; then runcmd="$runcmd --insert_only"; fi
if [ $cmd = "replace_into" ] ; then runcmd="replace_into.py --use_replace_into"; fi
if [ $cmd = "insert_ignore" ] ; then runcmd="replace_into.py"; fi
./$runcmd $args >>$runfile 2>&1
exitcode=$?
echo `date` $cmd complete $exitcode >>$runfile
if [ $exitcode != 0 ] ; then testresult="FAIL"; fi
popd
if [ $check != 0 -a $testresult = "PASS" ] ; then
echo `date` check table $tblname >>$runfile
mysql -S $mysqlsocket -u $mysqluser -D $dbname -e "check table $tblname" >>$runfile 2>&1
exitcode=$?
echo `date` check table $tblname $exitcode >>$runfile
if [ $exitcode != 0 ] ; then testresult="FAIL"; fi
fi
# commit results
if [ $commit != 0 ] ; then
if [ $cmd = "iibench" -a $insert_only != 0 ] ; then cmd="$cmd insert_only"; fi
svn add $runfile
retry svn commit -m \"$testresult $cmd $max_rows $dbname $mysqlbuild $mysqlserver `hostname`\" $runfile
fi
popd
if [ $testresult = "PASS" ] ; then exitcode=0; else exitcode=1; fi
exit $exitcode
scripts/run.mysql.tests.bash deleted 100755 → 0
View file @ f90c40ef
#!/usr/bin/env bash
# ident 4, no tabs
function usage() {
echo "run the tokudb mysql tests"
echo "--mysqlbuild=$mysqlbuild"
echo "--commit=$commit"
echo "--tests=$tests --engine=$engine"
}
function retry() {
local cmd
local retries
local exitcode
cmd=$*
let retries=0
while [ $retries -le 10 ] ; do
echo `date` $cmd
bash -c "$cmd"
exitcode=$?
echo `date` $cmd $exitcode $retries
let retries=retries+1
if [ $exitcode -eq 0 ] ; then break; fi
sleep 10
done
test $exitcode = 0
}
svnserver=https://svn.tokutek.com/tokudb
basedir=$HOME/svn.build
builddir=$basedir/mysql.build
mysqlbuild=
mysql_basedir=/usr/local/mysql
mysqlserver=`hostname`
commit=0
tests="*"
engine=""
parallel=auto
while [ $# -gt 0 ] ; do
arg=$1; shift
if [[ $arg =~ --(.*)=(.*) ]] ; then
eval ${BASH_REMATCH[1]}=${BASH_REMATCH[2]}
else
usage; exit 1
fi
done
if [[ $mysqlbuild =~ (.*)-(tokudb\-.*)-(linux)-(x86_64) ]] ; then
mysql=${BASH_REMATCH[1]}
tokudb=${BASH_REMATCH[2]}
system=${BASH_REMATCH[3]}
arch=${BASH_REMATCH[4]}
else
echo $mysqlbuild is not a tokudb build
fi
if [ -d $mysql_basedir/lib/mysql ] ; then
export LD_LIBRARY_PATH=$mysql_basedir/lib/mysql
fi
# update the build directory
if [ ! -d $basedir ] ; then mkdir $basedir ; fi
pushd $basedir
if [ $? != 0 ] ; then exit 1; fi
if [ ! -d $builddir ] ; then mkdir $builddir; fi
# make the subversion directory that will hold the test results
date=`date +%Y%m%d`
testresultsdir=$builddir/$date
pushd $builddir
if [ $? = 0 ] ; then
while [ ! -d $date ] ; do
svn mkdir $svnserver/mysql.build/$date -m ""
svn checkout -q $svnserver/mysql.build/$date
if [ $? -ne 0 ] ; then rm -rf $date; fi
done
popd
fi
# generate a trace file name
if [ -z $engine ] ; then
tracefile=mysql-test-$mysqlbuild-$mysqlserver
else
tracefile=mysql-engine-$engine-$mysqlbuild-$mysqlserver
fi
echo >$testresultsdir/$tracefile
if [ -z $engine ] ; then
# run all test suites including main
teststorun_original="main"
teststorun_tokudb=""
pushd $mysql_basedir/mysql-test/suite
if [ $? = 0 ] ; then
for t in $tests ; do
if [[ $t =~ .*\.xfail$ ]] ; then continue; fi
if [ $t = "perfschema_stress" ] ; then continue; fi
if [ $t = "large_tests" ] ; then continue; fi
if [ $t = "pbxt" ] ; then continue; fi
if [ -d $t/t ] ; then
if [[ $t =~ tokudb* ]] ; then
if [ -z $teststorun_tokudb ] ; then teststorun_tokudb="$t" ; else teststorun_tokudb="$teststorun_tokudb,$t"; fi
else
teststorun_original="$teststorun_original,$t";
fi
fi
done
popd
fi
# run the tests
pushd $mysql_basedir/mysql-test
if [ $? = 0 ] ; then
if [[ $mysqlbuild =~ tokudb ]] ; then
# run standard tests
if [[ $mysqlbuild =~ 5\\.5 ]] ; then
./mysql-test-run.pl --suite=$teststorun_original --big-test --max-test-fail=0 --force --retry=1 --testcase-timeout=60 \
--mysqld=--default-storage-engine=myisam --mysqld=--sql-mode="" \
--mysqld=--loose-tokudb_debug=3072 \
--parallel=$parallel >>$testresultsdir/$tracefile 2>&1
else
./mysql-test-run.pl --suite=$teststorun_original --big-test --max-test-fail=0 --force --retry=1 --testcase-timeout=60 \
--mysqld=--loose-tokudb_debug=3072 \
--parallel=$parallel >>$testresultsdir/$tracefile 2>&1
fi
# run tokudb tests
./mysql-test-run.pl --suite=$teststorun_tokudb --big-test --max-test-fail=0 --force --retry=1 --testcase-timeout=60 \
--mysqld=--loose-tokudb_debug=3072 \
--parallel=$parallel >>$testresultsdir/$tracefile 2>&1
# setup for engines tests
engine="tokudb"
else
./mysql-test-run.pl --suite=$teststorun_original --big-test --max-test-fail=0 --force --retry=1 --testcase-timeout=60 \
--parallel=$parallel >>$testresultsdir/$tracefile 2>&1
fi
popd
fi
fi
if [ ! -z $engine ] ; then
teststorun="engines/funcs,engines/iuds"
pushd $mysql_basedir/mysql-test
if [ $? = 0 ] ; then
if [[ $mysqlbuild =~ 5\\.6 ]] ; then
./mysql-test-run.pl --suite=$teststorun --force --retry-failure=0 --max-test-fail=0 --nowarnings --testcase-timeout=60 \
--mysqld=--default-storage-engine=$engine --mysqld=--default-tmp-storage-engine=$engine \
--parallel=$parallel >>$testresultsdir/$tracefile 2>&1
else
./mysql-test-run.pl --suite=$teststorun --force --retry-failure=0 --max-test-fail=0 --nowarnings --testcase-timeout=60 \
--mysqld=--default-storage-engine=$engine \
--parallel=$parallel >>$testresultsdir/$tracefile 2>&1
fi
popd
fi
fi
# summarize the results
let tests_failed=0
let tests_passed=0
while read line ; do
if [[ "$line" =~ (Completed|Timeout):\ Failed\ ([0-9]+)\/([0-9]+) ]] ; then
# failed[2]/total[3]
let tests_failed=tests_failed+${BASH_REMATCH[2]}
let tests_passed=tests_passed+${BASH_REMATCH[3]}-${BASH_REMATCH[2]}
elif [[ "$line" =~ Completed:\ All\ ([0-9]+)\ tests ]] ; then
# passed[1]
let tests_passed=tests_passed+${BASH_REMATCH[1]}
fi
done <$testresultsdir/$tracefile
# commit the results
if [ $tests_failed = 0 ] ; then
testresult="PASS=$tests_passed"
else
testresult="FAIL=$tests_failed PASS=$tests_passed"
fi
pushd $testresultsdir
if [ $? = 0 ] ; then
if [ $commit != 0 ] ; then
svn add $tracefile
if [[ $tracefile =~ "mysql-test" ]] ; then test=mysql-test; else test=mysql-engine-$engine; fi
retry svn commit -m \"$testresult $test $mysqlbuild $mysqlserver\" $tracefile
fi
popd
fi
popd # $basedir
if [[ $testresult =~ "PASS" ]] ; then exitcode=0; else exitcode=1; fi
exit $exitcode
scripts/run.sql.bench.bash deleted 100755 → 0
View file @ f90c40ef
#!/usr/bin/env bash
function usage() {
echo "run the sql bench tests"
echo "--mysqlbuild=$mysqlbuild"
echo "--commit=$commit"
}
function retry() {
local cmd
local retries
local exitcode
cmd=$*
let retries=0
while [ $retries -le 10 ] ; do
echo `date` $cmd
bash -c "$cmd"
exitcode=$?
echo `date` $cmd $exitcode $retries
let retries=retries+1
if [ $exitcode -eq 0 ] ; then break; fi
sleep 10
done
test $exitcode = 0
}
svnserver=https://svn.tokutek.com/tokudb
basedir=$HOME/svn.build
builddir=$basedir/mysql.build
mysqlbuild=
mysqlserver=`hostname`
commit=0
engine=tokudb
socket=/tmp/mysql.sock
system=`uname -s | tr [:upper:] [:lower:]`
arch=`uname -m | tr [:upper:] [:lower:]`
# parse the command line
while [ $# -gt 0 ] ; do
arg=$1; shift
if [[ $arg =~ --(.*)=(.*) ]] ; then
eval ${BASH_REMATCH[1]}=${BASH_REMATCH[2]}
else
usage; exit 1
fi
done
if [[ $mysqlbuild =~ (.*)-(tokudb-.*)-(linux)-(x86_64) ]] ; then
mysql=${BASH_REMATCH[1]}
tokudb=${BASH_REMATCH[2]}
system=${BASH_REMATCH[3]}
arch=${BASH_REMATCH[4]}
else
echo $mysqlbuild is not a tokudb build
fi
# goto the base directory
if [ ! -d $basedir ] ; then mkdir $basedir; fi
pushd $basedir
# update the build directory
if [ ! -d $builddir ] ; then mkdir $builddir; fi
date=`date +%Y%m%d`
testresultsdir=$builddir/$date
pushd $builddir
while [ ! -d $date ] ; do
svn mkdir $svnserver/mysql.build/$date -m ""
svn checkout -q $svnserver/mysql.build/$date
if [ $? -ne 0 ] ; then rm -rf $date; fi
done
popd
# run the tests
pushd /usr/local/mysql/sql-bench
tracefile=sql-bench-$engine-$mysqlbuild-$mysqlserver.trace
summaryfile=sql-bench-$engine-$mysqlbuild-$mysqlserver.summary
function mydate() {
date +"%Y%m%d %H:%M:%S"
}
function runtests() {
testargs=$*
for testname in test-* ; do
chmod +x ./$testname
echo `mydate` $testname $testargs
./$testname $testargs
exitcode=$?
echo `mydate`
if [ $exitcode != 0 ] ; then
# assume that the test failure due to a crash. allow mysqld to restart.
sleep 60
fi
done
}
>$testresultsdir/$tracefile
runtests --create-options=engine=$engine --socket=$socket --verbose --small-test >>$testresultsdir/$tracefile 2>&1
runtests --create-options=engine=$engine --socket=$socket --verbose --small-test --fast >>$testresultsdir/$tracefile 2>&1
runtests --create-options=engine=$engine --socket=$socket --verbose >>$testresultsdir/$tracefile 2>&1
runtests --create-options=engine=$engine --socket=$socket --verbose --fast >>$testresultsdir/$tracefile 2>&1
runtests --create-options=engine=$engine --socket=$socket --verbose --fast --fast-insert >>$testresultsdir/$tracefile 2>&1
runtests --create-options=engine=$engine --socket=$socket --verbose --fast --lock-tables >>$testresultsdir/$tracefile 2>&1
popd
# summarize the results
while read l ; do
if [[ $l =~ ^([0-9]{8}\ [0-9]{2}:[0-9]{2}:[0-9]{2})(.*)$ ]] ; then
t=${BASH_REMATCH[1]}
cmd=${BASH_REMATCH[2]}
if [ -z "$cmd" ] ; then
let duration=$(date -d "$t" +%s)-$(date -d "$tlast" +%s)
printf "%4s %s %8d %s\n" "$status" "$tlast" "$duration" "$cmdlast"
else
cmdlast=$cmd
tlast=$t
status=PASS
fi
else
if [[ $l =~ Got\ error|Died ]] ; then
status=FAIL
fi
fi
done <$testresultsdir/$tracefile >$testresultsdir/$summaryfile
testresult=""
pf=`mktemp`
egrep "^PASS" $testresultsdir/$summaryfile >$pf 2>&1
if [ $? -eq 0 ] ; then testresult="PASS=`cat $pf | wc -l` $testresult"; fi
egrep "^FAIL" $testresultsdir/$summaryfile >$pf 2>&1
if [ $? -eq 0 ] ; then testresult="FAIL=`cat $pf | wc -l` $testresult"; fi
rm $pf
if [ "$testresult" = "" ] ; then testresult="?"; fi
# commit the results
pushd $testresultsdir
if [ $commit != 0 ] ; then
svn add $tracefile $summaryfile
retry svn commit -m \"$testresult sql-bench $mysqlbuild $mysqlserver\" $tracefile $summaryfile
fi
popd
popd
if [[ $testresult =~ "PASS" ]] ; then exitcode=0; else exitcode=1; fi
exit $exitcode
scripts/run.tpch.bash deleted 100755 → 0
View file @ f90c40ef
This diff is collapsed.
scripts/setup.mysql.bash deleted 100755 → 0
View file @ f90c40ef
#!/usr/bin/env bash
function usage() {
echo "setup.mysql.bash"
echo "--mysqlbuild=$mysqlbuild --shutdown=$shutdown --install=$install --startup=$startup"
}
function download_file() {
local file=$1
s3get $s3bucket $file $file
}
function download_tarball() {
local tarball=$1
if [ ! -f $tarball ] ; then
download_file $tarball
if [ $? -ne 0 ] ; then test 0 = 1; return; fi
fi
if [ ! -f $tarball.md5 ] ; then
download_file $tarball.md5
if [ $? -ne 0 ] ; then test 0 = 1; return; fi
fi
}
function install_tarball() {
local basedir=$1; local tarball=$2
tar -x -z -f $basedir/$tarball
if [ $? -ne 0 ] ; then test 0 = 1; return; fi
}
function check_md5() {
local tarball=$1
md5sum --check $tarball.md5
if [ $? -ne 0 ] ; then
# support jacksum md5 output which is almost the same as md5sum
diff -b <(cat $tarball.md5) <(md5sum $tarball)
if [ $? -ne 0 ] ; then test 0 = 1; return; fi
fi
}
mysqlbuild=
shutdown=1
install=1
startup=1
s3bucket=tokutek-mysql-build
sleeptime=60
builtins="mysqlbuild shutdown install startup s3bucket sleeptime"
mysqld_args="--user=mysql --core-file --core-file-size=unlimited"
sudo=/usr/bin/sudo
defaultsfile=""
if [ -f /etc/$(whoami).my.cnf ] ; then
defaultsfile=/etc/$(whoami).my.cnf
fi
function is_builtin() {
local v=$1; shift
local x
for x in $* ; do
if [ $v = $x ] ; then echo 1; return; fi
done
echo 0
}
while [ $# -gt 0 ] ; do
arg=$1; shift
if [ $arg = "--help" -o $arg = "-h" -o $arg = "-?" ] ; then
usage; exit 1
elif [[ $arg =~ --(.*)=(.*) ]] ; then
r=$(is_builtin ${BASH_REMATCH[1]} $builtins)
if [ $r = 1 ] ; then
eval ${BASH_REMATCH[1]}=${BASH_REMATCH[2]}
else
mysqld_args="$mysqld_args $arg"
fi
else
mysqld_args="$mysqld_args $arg"
fi
done
if [ -d /data/mysql/tmp ] ; then mysqld_args="$mysqld_args --tmpdir=/data/mysql/tmp"; fi
if [[ $mysqlbuild =~ (.*)-(tokudb\-.*)-(linux)-(x86_64) ]] ; then
mysql=${BASH_REMATCH[1]}
tokudb=${BASH_REMATCH[2]}
system=${BASH_REMATCH[3]}
arch=${BASH_REMATCH[4]}
else
echo $mysqlbuild is not a tokudb build
fi
if [ ! -d downloads ] ; then mkdir downloads; fi
pushd downloads
if [ $? != 0 ] ; then exit 1; fi
basedir=$PWD
mysqltarball=$mysqlbuild.tar.gz
# get the tarball
download_tarball $mysqltarball
if [ $? -ne 0 ] ; then exit 1; fi
# check the md5 sum
check_md5 $mysqltarball
if [ $? -ne 0 ] ; then exit 1; fi
tokudbtarball=""
if [[ $mysqltarball =~ ^(Percona-Server.*)\.(Linux\.x86_64.*)$ ]] ; then
tar tzf $mysqltarball | egrep ha_tokudb.so >/dev/null 2>&1
if [ $? -ne 0 ] ; then
tokudbtarball=${BASH_REMATCH[1]}.TokuDB.${BASH_REMATCH[2]}
download_tarball $tokudbtarball
if [ $? -ne 0 ] ; then exit 1; fi
check_md5 $tokudbtarball
if [ $? -ne 0 ] ; then exit 1; fi
fi
fi
# set ldpath
ldpath=""
if [ -d /usr/local/gcc-4.7/lib64 ] ; then
echo skip ldpath="export LD_LIBRARY_PATH=/usr/local/gcc-4.7/lib64:\$LD_LIBRARY_PATH;"
fi
# shutdown mysql
if [ $shutdown -ne 0 ] ; then
if [ -x /etc/init.d/mysql ] ; then
$sudo setsid /etc/init.d/mysql stop
else
/usr/local/mysql/bin/mysqladmin shutdown
fi
sleep $sleeptime
fi
pushd /usr/local
if [ $? = 0 ] ; then
rm mysql
popd
fi
# install the release
pushd /usr/local/mysqls 2>/dev/null
if [ $? = 0 ] ; then
mysqldir=mysqls/$mysqlbuild
else
pushd /usr/local
if [ $? -ne 0 ] ; then exit 1; fi
mysqldir=$mysqlbuild
fi
if [ ! -d $mysqlbuild ] || [ $install -ne 0 ] ; then
rm mysql
if [ -d $mysqlbuild ] ; then $sudo rm -rf $mysqlbuild; fi
install_tarball $basedir $mysqltarball
if [ $? -ne 0 ] ; then exit 1; fi
if [ $tokudbtarball ] ; then
install_tarball $basedir $tokudbtarball
if [ $? -ne 0 ] ; then exit 1; fi
fi
ln -s $mysqldir /usr/local/mysql
if [ $? -ne 0 ] ; then exit 1; fi
ln -s $mysqldir /usr/local/$mysqlbuild
if [ $? -ne 0 ] ; then exit 1; fi
installdb=$mysqlbuild/bin/mysql_install_db
if [ ! -f $installdb ] ; then
installdb=$mysqlbuild/scripts/mysql_install_db
fi
$sudo chown -R mysql $mysqlbuild/data
$sudo chgrp -R mysql $mysqlbuild/data
# 5.6 debug build needs this
if [ ! -f $mysqlbuild/bin/mysqld ] && [ -f $mysqlbuild/bin/mysqld-debug ] ; then
ln $mysqlbuild/bin/mysqld-debug $mysqlbuild/bin/mysqld
fi
if [ -z "$defaultsfile" ] ; then
default_arg=""
else
default_arg="--defaults-file=$defaultsfile"
fi
$sudo bash -c "$ldpath $installdb $default_arg --user=mysql --basedir=$PWD/$mysqlbuild --datadir=$PWD/$mysqlbuild/data"
if [ $? -ne 0 ] ; then exit 1; fi
else
# create link
rm /usr/local/mysql
ln -s $mysqldir /usr/local/mysql
if [ $? -ne 0 ] ; then exit 1; fi
rm /usr/local/$mysqlbuild
ln -s $mysqldir /usr/local/$mysqlbuild
if [ $? -ne 0 ] ; then exit 1; fi
fi
popd
# start mysql
if [ $startup -ne 0 ] ; then
ulimit -a
# increase the open file limit
ulimit -n 10240
exitcode=$?
echo ulimit -n 10240 exitcode $exitcode
if [ -x /etc/init.d/mysql ] ; then
$sudo setsid /etc/init.d/mysql start
else
if [ -z "$defaultsfile" ] ; then
default_arg=""
else
default_arg="--defaults-file=$defaultsfile"
fi
j=/usr/local/mysql/lib/mysql/libjemalloc.so
if [ -f $j ] ; then
default_arg="$default_arg --malloc-lib=$j"
fi
$sudo -b bash -c "$ldpath /usr/local/mysql/bin/mysqld_safe $default_arg $mysqld_args" >/dev/null 2>&1 &
fi
sleep $sleeptime
# add mysql grants
/usr/local/mysql/bin/mysql -u root -e "grant all on *.* to tokubuild@localhost"
/usr/local/mysql/bin/mysql -u root -e "grant all on *.* to 'ec2-user'@localhost"
fi
popd
exit 0
scripts/test.mysql.bash deleted 100755 → 0
View file @ f90c40ef
#!/usr/bin/env bash
function usage() {
echo "run the mysql tests"
echo "--mysqlbuild=$mysqlbuild --tests=$tests"
}
function expand() {
echo $* | tr ,: " "
}
mysqlbuild=
mysqlsocket=/tmp/mysql.sock
gearmandir=/usr/local/gearmand-1.1.6
gearmandhost=localhost
system=$(uname -s | tr [:upper:] [:lower:])
arch=$(uname -m | tr [:upper:] [:lower:])
tests=run.mysql.tests.bash
while [ $# -gt 0 ] ; do
arg=$1; shift
if [[ $arg =~ --(.*)=(.*) ]] ; then
eval ${BASH_REMATCH[1]}=${BASH_REMATCH[2]}
else
usage; exit 1;
fi
done
if [ -z $mysqlbuild ] ; then exit 1; fi
for testname in $(expand $tests) ; do
if [ $testname = "run.mysql.tests.bash" ] ; then
run_mysqld=0
else
run_mysqld=1
fi
if [ $run_mysqld = 0 ] ; then
setupextra="--shutdown=1 --install=1 --startup=0"
else
setupextra="--shutdown=1 --install=1 --startup=1"
fi
echo "echo \$(date) $mysqlbuild >>/tmp/$(whoami).$testname.trace 2>&1; \
\$HOME/bin/setup.mysql.bash --mysqlbuild=$mysqlbuild $setupextra >>/tmp/$(whoami).$testname.trace 2>&1; \
testexitcode=\$?; \
echo \$(date) $mysqlbuild \$testexitcode >>/tmp/$(whoami).$testname.trace 2>&1; \
if [ \$testexitcode -ne 0 ] ; then exit 1; fi; \
\$HOME/bin/$testname --mysqlbuild=$mysqlbuild --commit=1 >>/tmp/$(whoami).$testname.trace 2>&1; \
if [ $run_mysqld != 0 ] ; then mysqladmin -S$mysqlsocket shutdown; fi" | $gearmandir/bin/gearman -b -f mysql-test-$system-$arch -h $gearmandhost -p 4730
done
exit 0
scripts/testbuildfromsrc.bash deleted 100644 → 0
View file @ f90c40ef
#!/usr/bin/env bash
# for all source tarballs and their coresponding md5 files, build a binary release tarball
system=$(uname -s|tr [:upper:] [:lower:])
arch=$(uname -m)
function expand() {
echo $* | tr ,: " "
}
for f in *.md5; do
if [[ $f =~ (.*).tar.gz.md5 ]] ; then
mysqlsrc=${BASH_REMATCH[1]}
else
exit 1
fi
if [ -d $mysqlsrc ] ; then continue; fi
md5sum --check $mysqlsrc.tar.gz.md5
if [ $? != 0 ] ; then exit 1; fi
tar xzf $mysqlsrc.tar.gz
if [ $? != 0 ] ; then exit 1; fi
mkdir $mysqlsrc/build.RelWithDebInfo
pushd $mysqlsrc/build.RelWithDebInfo
if [ $? != 0 ] ; then exit 1; fi
cmake -D BUILD_CONFIG=mysql_release -D CMAKE_BUILD_TYPE=RelWithDebInfo -D BUILD_TESTING=OFF ..
if [ $? != 0 ] ; then exit 1; fi
make -j4 package
if [ $? != 0 ] ; then exit 1; fi
if [ ! -f $mysqlsrc-$system-$arch.tar.gz ] ; then exit 1; fi
popd
done
scripts/testsandbox.bash deleted 100644 → 0
View file @ f90c40ef
#!/usr/bin/env bash
# for all tokudb binary tarballs, verify that we can create and run the tarball using the MySQL sandbox.
function expand() {
echo $* | tr ,: " "
}
let n=0
for f in *.md5; do
if [[ $f =~ (.*).tar.gz.md5 ]] ; then
mysqlbuild=${BASH_REMATCH[1]}
else
exit 1
fi
md5sum --check $f
if [ $? != 0 ] ; then exit 1; fi
make_sandbox --add_prefix=test$n- $mysqlbuild.tar.gz -- --sandbox_directory=test$n
if [ $? != 0 ] ; then exit 1; fi
pushd $HOME/sandboxes
if [ $? = 0 ] ; then
./use_all 'show engines'
./use_all 'create table test.t (a int primary key, b bigint, c varchar(256), d blob(500000), clustering key(b))'
./use_all 'show create table test.t'
./stop_all
popd
fi
let n=n+1
done
scripts/tpch.readme deleted 100644 → 0
View file @ f90c40ef
TPCH is an industry standard data warehouse benchmark. We use TPCH databases to test the TokuDB loader.
The run.tpch.bash script loads a TPCH database at a given scale factor into TokuDB. It then uses the SQL
CHECK TABLE statement to verify the fractal tree invariants. Finally, it dumps the database and compares
with a precomputed dump of the database from InnoDB.
Here are some TPCH databases dumped from InnoDB. These dumps are used to compare with TPCH data loaded
into TokuDB.
$ s3ls tokutek-mysql-data -l --prefix=tpch
2010-08-16T21:21:10.000Z 1073741824 tpch10G_data_dump.tar.0
2010-08-16T21:38:45.000Z 1073741824 tpch10G_data_dump.tar.1
2010-08-16T21:56:43.000Z 1073741824 tpch10G_data_dump.tar.2
2010-08-16T22:14:49.000Z 1073741824 tpch10G_data_dump.tar.3
2010-08-16T22:32:38.000Z 1073741824 tpch10G_data_dump.tar.4
2010-08-16T22:51:04.000Z 1073741824 tpch10G_data_dump.tar.5
2010-08-16T23:08:51.000Z 91262976 tpch10G_data_dump.tar.6
2010-08-16T23:10:21.000Z 654 tpch10G_data_dump.tar.xml
2010-08-12T17:45:09.000Z 633579520 tpch1G_data_dump.tar
2010-08-12T17:56:30.000Z 160 tpch1G_data_dump.tar.xml
2010-08-06T13:57:51.000Z 633610240 tpch1G_data_dump_innodb.tar
2010-08-06T14:07:09.000Z 174 tpch1G_data_dump_innodb.tar.xml
2010-11-28T12:20:58.000Z 886 tpch30G_data_dump.tar.xml
2010-09-14T19:16:30.000Z 1073741824 tpch30G_dump_data.tar.0
2010-09-14T19:40:02.000Z 1073741824 tpch30G_dump_data.tar.1
2010-09-14T20:12:22.000Z 1073741824 tpch30G_dump_data.tar.2
2010-09-14T20:45:23.000Z 1073741824 tpch30G_dump_data.tar.3
2010-09-14T21:14:07.000Z 1073741824 tpch30G_dump_data.tar.4
2010-09-14T21:37:54.000Z 1073741824 tpch30G_dump_data.tar.5
2010-09-14T21:57:02.000Z 1073741824 tpch30G_dump_data.tar.6
2010-09-14T22:16:59.000Z 1073741824 tpch30G_dump_data.tar.7
2010-09-14T22:36:22.000Z 1073741824 tpch30G_dump_data.tar.8
2010-09-14T22:55:25.000Z 382511104 tpch30G_dump_data.tar.9
2010-09-14T23:02:04.000Z 886 tpch30G_dump_data.tar.xml
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7