2014年9月30日火曜日

SoftLayerに冗長化NFSサーバを(その10)

File Storage (Consistent Performance) という NFS ベースのサービスが始まりました。
本稿では、こちらの検証結果を述べます。

名前からすると、不変のパフォーマンス、首尾一貫したパフォーマンスが特徴のファイルストレージということのようです。当ブログでは、基本的にパフォーマンス測定結果の掲載は見合わせるつもりでした。テスト方法を少し変えるだけで結果が全く異なってしまうことも珍しくはなく、公平な結果を掲載する自信がなかったからです。今回はパフォーマンス測定をしないことには話が進まないので例外とします。推測するな、計測せよ、との至言もありますが、今回は、計測するまでもなく、File Storage (Consistent Performance) についての結論は、私的にはぼぼ出ています。出ていますが、説得力を持たせるために計測します。計測前に予断を持つことはよくないのですが、あまりにも予想が立ってしまいます。
最近のストレージでは、2GB くらいのワークロードだと簡単にキャッシュに載ってしまいます。OS からの直接書き込み指示をどこまで守ってくれるのかも怪しいです。そこで、15GB のワークロードを5回繰り返しました。予算の都合もありますが、時間もかなりかかってしまう(1週間、フル稼働でした)ので、このあたりで妥協しました。どこまで現実を反映しているかは読者の判断にお任せします。
Linux 上で fio を動かした結果をそのまま掲載します。fio でファイル名を指定するのでなくディレクトリ名を指定しておけば、全てのジョブが別の領域 (ファイル) で仕事をしてワークロードの範囲を広げてくれます。詳細は手順を後述する際に設定ファイルで示します。

予算の関係で、File Storage (Consistent Performance) は一番安いものを発注しました。
File Storage (Consistent Performance) の選び方としては、まずサイズを決め、そのサイズに対応した IOPS を指定します。単価は一貫していて、月額課金のみ、容量単価 $0.1/GBパフォーマンス単価 $0.12/IOPS となっています。
File Storage (Consistent Performance) として用意されているラインナップは以下の通りです。


   20GB :  100 - 1000 IOPS
   40GB :  100 - 2000 IOPS
   80GB :  100 - 4000 IOPS
  100GB :  100 - 6000 IOPS
  250GB :  100 - 6000 IOPS
  500GB :  100 - 6000 IOPS
 1000GB :  100 - 6000 IOPS
 2000GB :  200 - 6000 IOPS
 4000GB :  300 - 6000 IOPS
 8000GB :  500 - 6000 IOPS
12000GB : 1000 - 6000 IOPS


IOPS は Input Output Per Second の略で、そのストレージが1秒間に処理できる I/O リクエスト数のことです。ストレージ性能の指標としてはこの値が一番重要です。全く別の概念で、帯域 (Bandwidth) と呼ばれるものがあります。こちらは単位時間当たり最大でどれだけのデータ量を流すことができるかを表しています。
今回の主役である File Storage (Consistent Performance) はネットワーク経由で利用するので、ネットワーク帯域を超えることはありません。1Gbps の NIC を選択した場合、どれだけ速いストレージと接続したとしても帯域の上限は 1Gbps となります。うまく冗長化して帯域を束ねることができれば、2Gbps が上限となります。これらの帯域は保証されたものではなく、理論的な上限となります。
SoftLayer では問題ないですが、束ね方も気を付けないと却って遅くなります。ボンディングのラウンドロビンでは、本数を増やすほど速度が落ちていき、1本のときが一番速かったという経験があります。
話を戻します。File Storage (Consistent Performance) では、帯域については何ら述べられていません。帯域保証型 NAS と呼ぶのは明らかに誤りです。RHEL の場合ですと、最小 I/O 単位が 4KB、最大 I/O 単位が 1MB となります。6000 IOPS のストレージを選び、1MB 単位でアクセスすれば 6Gbps の帯域が保証される、ということではありえません。帯域保証はされていないので当然のことです。
プロトコルのオーバヘッドも考えるとネットワーク越しのデータ転送で 100MB/s を超えることはまずないはずです。HP 社の実験室で HP-UX を使っている時に FTP 通信で 110MB/s を計測したことがありますが、これ以外で Gigabit Ethernet を単体利用して 100MB/s を超えた経験はありません。
また、あえてシーケンシャルアクセスさせている場合を除けば、基本的には 4KB もしくは 8KB でのランダムアクセスしか発生しない、くらいの考えでいた方が現実に即していると思います。
6000 IOPS の性能で 4KB のランダムアクセスを処理するには、24MB/s の帯域があれば理論的には十分です。実際にはプロトコルのオーバヘッドやエラー対策もあるので、2~3倍の値を見込む必要があるかと思われます。ジャンボフレームに対応していない状況では最小単位のアクセスの場合ですらパケット(フレーム)の分割も発生します。このあたりの数字を見ていくと、6000 IOPS が上限となっている理由が見えてきます。
ストレージ性能の上限はまだまだ上にありますが、あえて上限を低く抑えることで公平にリソースを配分できるようにしているのだと思われます。計測中に IOPS の平均値を眺めていましたが、100 IOPS のものはぴったり 100 IOPS という性能が出ていました。たまに 99 IOPS となったり、 101 IOPS となることもありましたが、誤差の範囲です。
では、100 IOPS の File Storage (Consistent Performance) において、1MBブロックのシーケンシャルアクセスでどの程度の性能が出るのかですが、6~7 IOPS ほどしか出ませんでした。100 IOPS を保証しているわけでもないようです。6400KB/s ほどの帯域しか使っていませんでした。ネットワークがボトルネックとなっているわけではないので、帯域保証型でないだけでなく、IOPS 保証型ですらないです。もっとも、1ブロック最大 64KB が仕様だ、と言ってしまえば、IOPS 保証があるとの解釈もできます。
普遍のパフォーマンス、首尾一貫したパフォーマンスというのは、性能の上限のことをさしているだけのようです。この点は予想よりもさらに下を行ってしまうという悲しい結果となってしまいました。やはり、計測は必要でした。

というわけで、20 GB、100 IOPS、$14/month の性能テストを実施しました。予算の問題もありますが、単純に IOPS 指定に比例すると考えてよいとの判断もあります。
安心して使える NFS で、$120/month を切るものは今のところこれしかありません。1TB、100 IOPS で $100+$12 = $112 です。高いパフォーマンスが保証されていることが売りなのではなく、パフォーマンスを制限すると安いということが売りなのだと思われます。1TB 全体の読み込みに46時間かけることに耐えられるのであればの話ですが。
iSCSI の話になりますが、クォーラムディスクのように、I/O は全くと言っていいほど発生しないが SCSI3 Persistent Reservation 機能が使えないと困る、という場合には最適なソリューションだと言えます。既存の Block Storage (Legacy iSCSI) の最安が $15/month だったので、少し安く調達できます。

File Storage (Consistent Performance) で一番高いものは、12TB、6000IOPS で $1200+$720 = $1920 となります。
8TB、2000IOPS だと $800+$240 = $1040 となります。
1TB、6000IOPS だと $100+$720 = $820 となります。

仮想サーバで冗長化 NFS サーバを用意した場合、読み取りで 7955 IOPS、書き込みで 2112 IOPS ほど出ました。
20GB 程度のものを用意する場合、2台で $107/month ほどになります。
最大 8TB 程度のものが用意できます。2台で $850/month ほどになります。こちらは今回テストしていません。金額の提示目的のみです。


HKG02 - Hong Kong - Asia
1 x 2.0 GHz Core
1 GB RAM
CentOS 6.x - Minimal Install (64 bit)
25 GB (SAN)
25 GB (SAN) $0.008/hour
1Gbps Public & Privaate Network Uplinks $0.040/hour

Total: $0.089/hour
->     $0.071/hour (≒$53.25/month)


HKG02 - Hong Kong - Asia
1 x 2.0 GHz Core
1 GB RAM
CentOS 6.x - Minimal Install (64 bit)
25 GB (SAN)
2000 GB (SAN) $0.157/hour
2000 GB (SAN) $0.157/hour
2000 GB (SAN) $0.157/hour
2000 GB (SAN) $0.157/hour

Total: $0.708/hour
->     $0.566/hour (≒$424.5/month)


仮想サーバで選択可能なポータブルストレージのラインナップは以下の通りです。Local と SAN を混在させることはできません。


Local first
  25GB $0.000/hour (≒$  0.00/month) $  0.00/month
 100GB $0.005/hour (≒$  3.75/month) $  4.00/month

Local second
  25GB $0.002/hour (≒$  1.50/month) $  1.00/month
 100GB $0.005/hour (≒$  3.75/month) $  4.00/month
 150GB $0.008/hour (≒$  6.00/month) $  6.00/month
 200GB $0.011/hour (≒$  8.25/month) $  8.00/month
 300GB $0.016/hour (≒$ 12.00/month) $ 12.00/month

SAN first
  25GB $0.000/hour (≒$  0.00/month) $  0.00/month
 100GB $0.014/hour (≒$ 10.50/month) $ 10.29/month

SAN second,third,fourth,fifth
  10GB $0.006/hour (≒$  4.50/month) $  4.57/month
  20GB $0.008/hour (≒$  6.00/month) $  5.71/month
  25GB $0.009/hour (≒$  6.75/month) $  6.50/month
  30GB $0.010/hour (≒$  7.50/month) $  6.86/month
  40GB $0.011/hour (≒$  8.25/month) $  8.00/month
  50GB $0.012/hour (≒$  9.00/month) $  9.14/month
  75GB $0.013/hour (≒$  9.75/month) $  9.71/month
 100GB $0.014/hour (≒$ 10.50/month) $ 10.29/month
 125GB $0.015/hour (≒$ 11.25/month) $ 10.86/month
 150GB $0.016/hour (≒$ 12.00/month) $ 11.43/month
 175GB $0.017/hour (≒$ 12.75/month) $ 12.00/month
 200GB $0.018/hour (≒$ 13.50/month) $ 12.57/month
 250GB $0.020/hour (≒$ 15.00/month) $ 14.29/month
 300GB $0.025/hour (≒$ 18.75/month) $ 18.57/month
 350GB $0.031/hour (≒$ 23.25/month) $ 22.86/month
 400GB $0.035/hour (≒$ 26.25/month) $ 25.71/month
 500GB $0.039/hour (≒$ 29.25/month) $ 28.57/month
 750GB $0.059/hour (≒$ 44.25/month) $ 42.86/month
1000GB $0.078/hour (≒$ 58.50/month) $ 57.14/month
1500GB $0.117/hour (≒$ 87.75/month) $ 85.71/month
2000GB $0.157/hour (≒$117.75/month) $114.29/month


物理サーバで冗長化 NFS サーバを用意した場合、読み取りで 30574 IOPS、書き込みで 18272 IOPS ほど出ました。
RAID コントローラ搭載可能 (4台以上のローカルディスク搭載可能が最低条件) で一番安い筐体を選び、一番安い SSD を2台にすると、サーバ2台で $610/month ほどになります。
SSD は、32GB $13.00/month のものを発注しましたが 900GB 超のものが供給されます。メモリも 8GB です。計算通り(? 期待通り)です。
SSD ではなく 4TB HDD を4台、というパターンも試したかったのですが、予算超過となってしまい見送りました。RAID10 で 8TB ですが、同じ容量を確保できる仮想サーバの結果が想定よりもはるかに良かったので、コストパフォーマンスでは完全に負けてしまいます。単純にパフォーマンスで比べても読み取りは負けるような気がしています。書き込みは RAID コントローラのキャッシュが飽和するまでは負けないとは思いますが、Adaptec のドライバーの実装を調べたことも無いので実測してみないと分からないです。
仮想サーバで冗長化 NFS サーバを構築する際の最大の欠点は同一ハイパーバイザ上で動く可能性があることでしたが、ソフトレイヤー探検隊隊長がラック番号の調べ方を掲載してくださいました(※)ので、死角はなくなったと考えています。DRBD の初期同期時に 2Gbps 出ていたので同一筐体で動いている可能性を考えていたのですが、SoftLayer の仮想サーバはかなり高性能なネットワーク環境が提供されていました。ジャンボフレーム対応していないことだけが問題ですが、これだけパフォーマンスがよければ本番環境用としても十分選択肢になり得るのではないかと思われます。

(※)ソフトレイヤー探検隊 第22回 仮想サーバーの動作環境を探ってみる (http://goo.gl/xQRJ9l)


Intel Xeon 3450 4 Cores 2.66GHz
Up to 4 drives
4 GB - 24 GB
$169.00/month

HKG02 - Hong Kong - Asia (Additional $20.00 monthly data center surcharge)
4 GB DDR3 Registered 1333
CentOS 6.x (64 bit)
32 of 32 GB SSD $13.00/month
32 of 32 GB SSD $13.00/month
Create RAID storage group $50.00/month
RAID 1
Size: 25
Linux Extra Partition
  /           10    GB
  /swap0       2    GB
  /boot        0.25 GB
  /disk0  Grow 1    GB
Create RAID storage group
RAID 1
Size: 2
Create RAID storage group
RAID 1
Size: 5
1Gbps Redundant Public & Privaate Network Uplinks $40.00/month

Total: $305/month


物理サーバで選択可能なローカルストレージのラインナップは以下の通りです。搭載可能台数は、最初に選択する筐体で決まります。36台搭載可能な筐体もあります。


 32GB SSD             $ 13.00
 50GB SSD             $ 20.00
 64GB SSD             $ 26.00
100GB SSD             $ 41.00
200GB SSD             $ 81.00
400GB SSD             $163.00
800GB SSD             $222.00

 73GB SA-SCSI 15K RPM $ 25.00
147GB SA-SCSI 15K RPM $ 42.00
300GB SA-SCSI 15K RPM $ 72.00
450GB SA-SCSI 15K RPM $ 99.00
600GB SA-SCSI 15K RPM $120.00

 73GB SA-SCSI 10K RPM $ 15.00
147GB SA-SCSI 10K RPM $ 28.00
300GB SA-SCSI 10K RPM $ 54.00
400GB SA-SCSI 10K RPM $ 85.00

 74GB SATA Raptor 10k $  5.00
150GB SATA Raptor 10k $ 10.00
300GB SATA Raptor 10k $ 15.00

 250GB SATA II        $ 11.00 (first HDD: $0.00)
 500GB SATA II        $ 21.00 (first HDD: $0.00)
 750GB SATA II        $ 32.00
1.00TB SATA II        $ 42.00
2.00TB SATA II        $ 60.00
3.00TB SATA III       $ 80.00
4.00TB SATA III       $100.00


テスト手順を記載します。

NFS クライアントとして、時間課金の物理サーバを用意しました。クライアント側がボトルネックとならないようにするためと、ジャンボフレームを有効化した状態でテストしたかったからです。File Storage (Consistent Performance) は当然ジャンボフレームに対応しています。仮想サーバは対応していません。

いつの間にか、時間課金の物理サーバでは、NIC の冗長化が選べなくなっていました。その代り、デプロイ時間が短くなったように思われます。残念に思っていたところ、ログインしてみてビックリ、ボンディング設定が入っていました。サービスしてくれているのかと喜んだのですが、リンクアップしていませんでした。冗長化設定が入っているだけで、冗長化はされていませんでした。

以下のスペックで発注しました。


Intel Xeon 1270 4 Cores 3.40GHz
Up to 2 drives
8 GB
$339.00/month
$0.465/hour

HKG02 - Hong Kong - Asia
8 GB DDR3 1333
CentOS 6.x (64 bit)
1TB SATA
1TB SATA
Linux Extra Partition
1Gbps Public & Privaate Network Uplinks $0.040/hour

Total: $0.505/hour (≒$378.75/month)


供給されたものは E3-1270 v3 @ 3.50GHz 8 Cores でした。その他は注文通りです。

クライアントの IP アドレスを Primary Address から Portable Address に変更します。File Storage (Consistent Performance) では、アクセス可能となるクライアントを事前に登録する必要があるのですが、対象ノードを見ており、対象ノードが使っているアドレスを見ている訳ではないようです。無事通信できます。Ping 監視の登録でもそうでしたが、Primary Address しか選べないようでいて、Portable Address に変更すると Portable Address も登録可能となります。このあたりはどのようなカラクリで制御されているのかわかりませんが、MAC アドレスではないかと思っています。そのうち、MAC アドレスをわざと変えてルータに通知してみようかと思ってはいますが、やってみたらどこかで報告することにします。
ここまで書いて、一度アンマウントし、再マウントしようとしたらアクセス権がない、というエラーが発生しました。一度アクセス可能なホスト登録から削除し、再登録しても改善しませんでした。監視設定を直さないと駄目なのかと思い、設定変更しようと思ったら監視設定用のタブがなくなっていました。Portable Address を Primary Address に戻すとマウントできました。
ここは仕様が固まっていないのか、Portable Address はサポートしない、という仕様になってしまったのか、不安が残るところです。
とりあえず、File Storage (Consistent Performance) については Primary Address にて性能測定を行います。
クライアントにおける準備手順は以下のとおりです。


##sudo /rescue/mk_portable_ip 10.110.88.59
sudo yum -y install nfs-utils
sudo /etc/init.d/rpcbind start
sudo /etc/init.d/nfslock start
sudo mkdir /backup
sudo yum --enablerepo=epel install fio


fio の設定ファイルを用意します。
最初の2つで性能を測定します。500MB のファイルに対し、4KB 単位でアクセスするジョブを30並列で動かします。5回繰り返します。このテストがメインのテストです。
次の2つはシーケンシャルアクセスの場合を念のため測定しています。
残りの2つはおまけで、1GB のファイルに対し、4KB 単位でアクセスするジョブを単体で動かします。5回繰り返します。


cat << EOF > randread4k30.fio
[randread4k30]
readwrite=randread
blocksize=4k
size=500m
directory=/backup/fio
direct=1
numjobs=30
loops=5
EOF
cat << EOF > randwrite4k30.fio
[randwrite4k30]
readwrite=randwrite
blocksize=4k
size=500m
directory=/backup/fio
direct=1
numjobs=30
loops=5
EOF
cat << EOF > seqread.fio
[seqread]
readwrite=read
blocksize=1m
size=10g
directory=/backup/fio
direct=1
loops=5
EOF
cat << EOF > seqwrite.fio
[seqwrite]
readwrite=write
blocksize=1m
size=10g
directory=/backup/fio
direct=1
loops=5
EOF
cat << EOF > randread4k.fio
[randread4k]
readwrite=randread
blocksize=4k
size=1g
directory=/backup/fio
direct=1
loops=5
EOF
cat << EOF > randwrite4k.fio
[randwrite4k]
readwrite=randwrite
blocksize=4k
size=1g
directory=/backup/fio
direct=1
loops=5
EOF


File Storage (Consistent Performance) に対する測定をします。


sudo mount -t nfs nfshkg0201d.service.softlayer.com:/SL01SV******_1 /backup
cat /proc/mounts | grep backupnfshkg0201d.service.softlayer.com:/SL01SV******_1/ /backup nfs4
rw,relatime,vers=4,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.110.69.10,minorversion=0,local_lock=none,addr=10.2.174.112 0 0

df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda2             917G  3.3G  867G   1% /
tmpfs                 3.9G     0  3.9G   0% /dev/shm
/dev/sda1             248M  101M  135M  43% /boot
nfshkg0201d.service.softlayer.com:/SL01SV******_1
                       20G  287M   20G   2% /backup

sudo mkdir -p /backup/fio && sudo fio randread4k30.fio  && sudo rm -rf /backup/fio
randread4k30: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=1
...
randread4k30: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=1
fio-2.0.13
Starting 30 processes
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
Jobs: 30 (f=30): [rrrrrrrrrrrrrrrrrrrrrrrrrrrrrr] [inf% done] [384K/0K/0K /s] [96 /0 /0  iops] [eta 1158050441d:07h:00
Jobs: 30 (f=30): [rrrrrrrrrrrrrrrrrrrrrrrrrrrrrr] [inf% done] [400K/0K/0K /s] [100 /0 /0  iops] [eta 1158050441d:07h:0
Jobs: 30 (f=30): [rrrrrrrrrrrrrrrrrrrrrrrrrrrrrr] [inf% done] [400K/0K/0K /s] [100 /0 /0  iops] [eta 1158050441d:07h:0
Jobs: 30 (f=30): [rrrrrrrrrrrrrrrrrrrrrrrrrrrrrr] [inf% done] [396K/0K/0K /s] [99 /0 /0  iops] [eta 1158050441d:07h:00
Jobs: 30 (f=30): [rrrrrrrrrrrrrrrrrrrrrrrrrrrrrr] [inf% done] [400K/0K/0K /s] [100 /0 /0  iops] [eta 1158050441d:07h:0
Jobs: 30 (f=30): [rrrrrrrrrrrrrrrrrrrrrrrrrrrrrr] [inf% done] [400K/0K/0K /s] [100 /0 /0  iops] [eta 1158050441d:07h:0
Jobs: 30 (f=30): [rrrrrrrrrrrrrrrrrrrrrrrrrrrrrr] [inf% done] [404K/0K/0K /s] [101 /0 /0  iops] [eta 1158050441d:07h:0
Jobs: 30 (f=30): [rrrrrrrrrrrrrrrrrrrrrrrrrrrrrr] [inf% done] [400K/0K/0K /s] [100 /0 /0  iops] [eta 1158050441d:07h:0
Jobs: 30 (f=30): [rrrrrrrrrrrrrrrrrrrrrrrrrrrrrr] [inf% done] [396K/0K/0K /s] [99 /0 /0  iops] [eta 1158050441d:07h:00
Jobs: 30 (f=30): [rrrrrrrrrrrrrrrrrrrrrrrrrrrrrr] [inf% done] [400K/0K/0K /s] [100 /0 /0  iops] [eta 1158050441d:07h:0
Jobs: 30 (f=30): [rrrrrrrrrrrrrrrrrrrrrrrrrrrrrr] [inf% done] [400K/0K/0K /s] [100 /0 /0  iops] [eta 1158050441d:07h:0
Jobs: 30 (f=30): [rrrrrrrrrrrrrrrrrrrrrrrrrrrrrr] [inf% done] [404K/0K/0K /s] [101 /0 /0  iops] [eta 1158050441d:07h:0
Jobs: 30 (f=30): [rrrrrrrrrrrrrrrrrrrrrrrrrrrrrr] [inf% done] [400K/0K/0K /s] [100 /0 /0  iops] [eta 1158050441d:07h:0
Jobs: 30 (f=30): [rrrrrrrrrrrrrrrrrrrrrrrrrrrrrr] [inf% done] [400K/0K/0K /s] [100 /0 /0  iops] [eta 1158050441d:07h:0
Jobs: 30 (f=30): [rrrrrrrrrrrrrrrrrrrrrrrrrrrrrr] [inf% done] [400K/0K/0K /s] [100 /0 /0  iops] [eta 1158050441d:06h:5
Jobs: 30 (f=30): [rrrrrrrrrrrrrrrrrrrrrrrrrrrrrr] [0.0% done] [400K/0K/0K /s] [100 /0 /0  iops] [eta 66d:15h:59m:42s]
Jobs: 1 (f=1): [______________r_______________] [100.0% done] [400K/0K/0K /s] [100 /0 /0  iops] [eta 00m:05s]   ]
randread4k30: (groupid=0, jobs=1): err= 0: pid=5278: Fri Sep 26 18:39:44 2014
  read : io=1941.1MB, bw=13708 B/s, iops=3 , runt=148531975msec
    clat (usec): min=422 , max=32661K, avg=298772.51, stdev=1584596.43
     lat (usec): min=422 , max=32661K, avg=298772.66, stdev=1584596.43
    clat percentiles (usec):
     |  1.00th=[  956],  5.00th=[ 5088], 10.00th=[ 9664], 20.00th=[10048],
     | 30.00th=[10944], 40.00th=[18304], 50.00th=[36096], 60.00th=[129536],
     | 70.00th=[220160], 80.00th=[301056], 90.00th=[481280], 95.00th=[684032],
     | 99.00th=[1019904], 99.50th=[16187392], 99.90th=[16711680], 99.95th=[16711680],
     | 99.99th=[16711680]
    bw (KB/s)  : min=    0, max=  408, per=6.62%, avg=26.50, stdev=46.62
    lat (usec) : 500=0.04%, 750=0.05%, 1000=1.14%
    lat (msec) : 2=1.50%, 4=1.27%, 10=12.18%, 20=26.39%, 50=9.09%
    lat (msec) : 100=5.60%, 250=15.29%, 500=18.17%, 750=5.06%, 1000=3.03%
    lat (msec) : 2000=0.39%, >=2000=0.80%
  cpu          : usr=0.00%, sys=0.00%, ctx=497817, majf=0, minf=26
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=497124/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=5279: Fri Sep 26 18:39:44 2014
  read : io=2500.0MB, bw=14552 B/s, iops=3 , runt=180131645msec
    clat (usec): min=435 , max=32552K, avg=281447.94, stdev=1527693.70
     lat (usec): min=435 , max=32552K, avg=281448.33, stdev=1527693.70
    clat percentiles (usec):
     |  1.00th=[  956],  5.00th=[ 5792], 10.00th=[ 9920], 20.00th=[10048],
     | 30.00th=[10944], 40.00th=[18304], 50.00th=[33536], 60.00th=[121344],
     | 70.00th=[209920], 80.00th=[301056], 90.00th=[460800], 95.00th=[659456],
     | 99.00th=[1003520], 99.50th=[16056320], 99.90th=[16711680], 99.95th=[16711680],
     | 99.99th=[16711680]
    bw (KB/s)  : min=    0, max=  411, per=6.86%, avg=27.42, stdev=48.99
    lat (usec) : 500=0.04%, 750=0.05%, 1000=1.27%
    lat (msec) : 2=1.46%, 4=0.94%, 10=11.06%, 20=27.65%, 50=9.40%
    lat (msec) : 100=5.85%, 250=16.66%, 500=16.91%, 750=4.79%, 1000=2.84%
    lat (msec) : 2000=0.36%, >=2000=0.72%
  cpu          : usr=0.00%, sys=0.00%, ctx=677640, majf=0, minf=26
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=640000/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=5280: Fri Sep 26 18:39:44 2014
  read : io=2500.0MB, bw=14625 B/s, iops=3 , runt=179232547msec
    clat (usec): min=422 , max=32651K, avg=280044.82, stdev=1529183.94
     lat (usec): min=422 , max=32651K, avg=280045.16, stdev=1529183.96
    clat percentiles (usec):
     |  1.00th=[  956],  5.00th=[ 5856], 10.00th=[ 9920], 20.00th=[10048],
     | 30.00th=[10944], 40.00th=[16512], 50.00th=[31104], 60.00th=[111104],
     | 70.00th=[201728], 80.00th=[301056], 90.00th=[452608], 95.00th=[651264],
     | 99.00th=[1011712], 99.50th=[16056320], 99.90th=[16711680], 99.95th=[16711680],
     | 99.99th=[16711680]
    bw (KB/s)  : min=    0, max=  408, per=6.99%, avg=27.98, stdev=49.76
    lat (usec) : 500=0.04%, 750=0.04%, 1000=1.26%
    lat (msec) : 2=1.44%, 4=0.93%, 10=11.69%, 20=27.85%, 50=9.40%
    lat (msec) : 100=5.77%, 250=16.65%, 500=16.38%, 750=4.69%, 1000=2.77%
    lat (msec) : 2000=0.35%, >=2000=0.74%
  cpu          : usr=0.00%, sys=0.00%, ctx=668355, majf=0, minf=26
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=640000/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=5281: Fri Sep 26 18:39:44 2014
  read : io=2500.0MB, bw=14622 B/s, iops=3 , runt=179277133msec
    clat (usec): min=440 , max=32593K, avg=280113.39, stdev=1541954.09
     lat (usec): min=441 , max=32593K, avg=280113.59, stdev=1541954.10
    clat percentiles (usec):
     |  1.00th=[  948],  5.00th=[ 5856], 10.00th=[ 9920], 20.00th=[10048],
     | 30.00th=[10944], 40.00th=[15424], 50.00th=[30848], 60.00th=[111104],
     | 70.00th=[199680], 80.00th=[301056], 90.00th=[452608], 95.00th=[659456],
     | 99.00th=[1011712], 99.50th=[16056320], 99.90th=[16711680], 99.95th=[16711680],
     | 99.99th=[16711680]
    bw (KB/s)  : min=    0, max=  408, per=7.04%, avg=28.16, stdev=50.71
    lat (usec) : 500=0.03%, 750=0.05%, 1000=1.31%
    lat (msec) : 2=1.43%, 4=0.91%, 10=11.65%, 20=28.23%, 50=9.38%
    lat (msec) : 100=5.69%, 250=16.44%, 500=16.35%, 750=4.65%, 1000=2.79%
    lat (msec) : 2000=0.36%, >=2000=0.73%
  cpu          : usr=0.00%, sys=0.00%, ctx=643722, majf=0, minf=26
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=640000/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=5282: Fri Sep 26 18:39:44 2014
  read : io=2500.0MB, bw=14675 B/s, iops=3 , runt=178629506msec
    clat (usec): min=446 , max=32621K, avg=279099.38, stdev=1511293.33
     lat (usec): min=446 , max=32621K, avg=279099.60, stdev=1511293.34
    clat percentiles (usec):
     |  1.00th=[  956],  5.00th=[ 5856], 10.00th=[ 9920], 20.00th=[10048],
     | 30.00th=[10944], 40.00th=[17792], 50.00th=[32640], 60.00th=[121344],
     | 70.00th=[201728], 80.00th=[301056], 90.00th=[452608], 95.00th=[659456],
     | 99.00th=[1003520], 99.50th=[16056320], 99.90th=[16711680], 99.95th=[16711680],
     | 99.99th=[16711680]
    bw (KB/s)  : min=    0, max=  408, per=6.91%, avg=27.64, stdev=49.54
    lat (usec) : 500=0.03%, 750=0.05%, 1000=1.31%
    lat (msec) : 2=1.46%, 4=0.91%, 10=11.13%, 20=27.79%, 50=9.17%
    lat (msec) : 100=5.78%, 250=16.98%, 500=16.82%, 750=4.66%, 1000=2.83%
    lat (msec) : 2000=0.35%, >=2000=0.72%
  cpu          : usr=0.00%, sys=0.00%, ctx=650140, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=640000/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=5283: Fri Sep 26 18:39:44 2014
  read : io=2500.0MB, bw=14848 B/s, iops=3 , runt=176545673msec
    clat (usec): min=451 , max=32615K, avg=275848.88, stdev=1511636.59
     lat (usec): min=451 , max=32615K, avg=275849.02, stdev=1511636.60
    clat percentiles (usec):
     |  1.00th=[  948],  5.00th=[ 5920], 10.00th=[ 9920], 20.00th=[10048],
     | 30.00th=[10944], 40.00th=[15424], 50.00th=[30336], 60.00th=[110080],
     | 70.00th=[199680], 80.00th=[292864], 90.00th=[452608], 95.00th=[651264],
     | 99.00th=[1003520], 99.50th=[16056320], 99.90th=[16711680], 99.95th=[16711680],
     | 99.99th=[16711680]
    bw (KB/s)  : min=    0, max=  408, per=7.06%, avg=28.26, stdev=50.32
    lat (usec) : 500=0.04%, 750=0.05%, 1000=1.25%
    lat (msec) : 2=1.43%, 4=0.92%, 10=11.96%, 20=28.00%, 50=9.50%
    lat (msec) : 100=5.72%, 250=16.40%, 500=16.28%, 750=4.61%, 1000=2.75%
    lat (msec) : 2000=0.36%, >=2000=0.72%
  cpu          : usr=0.00%, sys=0.00%, ctx=641159, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=640000/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=5284: Fri Sep 26 18:39:44 2014
  read : io=2500.0MB, bw=14731 B/s, iops=3 , runt=177950929msec
    clat (usec): min=447 , max=32671K, avg=278038.46, stdev=1516846.93
     lat (usec): min=447 , max=32671K, avg=278038.60, stdev=1516846.93
    clat percentiles (usec):
     |  1.00th=[  956],  5.00th=[ 5856], 10.00th=[ 9920], 20.00th=[10048],
     | 30.00th=[10944], 40.00th=[18304], 50.00th=[31104], 60.00th=[112128],
     | 70.00th=[199680], 80.00th=[301056], 90.00th=[452608], 95.00th=[651264],
     | 99.00th=[1003520], 99.50th=[16056320], 99.90th=[16711680], 99.95th=[16711680],
     | 99.99th=[16711680]
    bw (KB/s)  : min=    0, max=  492, per=6.99%, avg=27.98, stdev=49.41
    lat (usec) : 500=0.04%, 750=0.05%, 1000=1.24%
    lat (msec) : 2=1.42%, 4=0.95%, 10=11.51%, 20=27.57%, 50=9.72%
    lat (msec) : 100=5.82%, 250=16.79%, 500=16.38%, 750=4.68%, 1000=2.76%
    lat (msec) : 2000=0.35%, >=2000=0.72%
  cpu          : usr=0.00%, sys=0.00%, ctx=640025, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=640000/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=5285: Fri Sep 26 18:39:44 2014
  read : io=2500.0MB, bw=14786 B/s, iops=3 , runt=177283997msec
    clat (usec): min=420 , max=32613K, avg=276997.96, stdev=1510301.29
     lat (usec): min=420 , max=32613K, avg=276998.11, stdev=1510301.29
    clat percentiles (usec):
     |  1.00th=[  948],  5.00th=[ 5920], 10.00th=[ 9920], 20.00th=[10048],
     | 30.00th=[10944], 40.00th=[17792], 50.00th=[31104], 60.00th=[118272],
     | 70.00th=[201728], 80.00th=[301056], 90.00th=[452608], 95.00th=[651264],
     | 99.00th=[1003520], 99.50th=[16056320], 99.90th=[16711680], 99.95th=[16711680],
     | 99.99th=[16711680]
    bw (KB/s)  : min=    0, max=  408, per=6.98%, avg=27.92, stdev=49.65
    lat (usec) : 500=0.04%, 750=0.05%, 1000=1.26%
    lat (msec) : 2=1.45%, 4=0.93%, 10=11.55%, 20=27.66%, 50=9.50%
    lat (msec) : 100=5.79%, 250=16.85%, 500=16.44%, 750=4.69%, 1000=2.75%
    lat (msec) : 2000=0.35%, >=2000=0.71%
  cpu          : usr=0.00%, sys=0.00%, ctx=640800, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=640000/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=5286: Fri Sep 26 18:39:44 2014
  read : io=1890.8MB, bw=13875 B/s, iops=3 , runt=142890389msec
    clat (usec): min=457 , max=32688K, avg=295193.28, stdev=1567078.33
     lat (usec): min=458 , max=32688K, avg=295193.88, stdev=1567078.32
    clat percentiles (usec):
     |  1.00th=[  964],  5.00th=[ 4960], 10.00th=[ 9664], 20.00th=[10048],
     | 30.00th=[10944], 40.00th=[17280], 50.00th=[31616], 60.00th=[121344],
     | 70.00th=[218112], 80.00th=[301056], 90.00th=[477184], 95.00th=[684032],
     | 99.00th=[1019904], 99.50th=[16187392], 99.90th=[16711680], 99.95th=[16711680],
     | 99.99th=[16711680]
    bw (KB/s)  : min=    0, max=  408, per=6.67%, avg=26.69, stdev=46.62
    lat (usec) : 500=0.04%, 750=0.05%, 1000=1.07%
    lat (msec) : 2=1.82%, 4=1.22%, 10=12.06%, 20=26.61%, 50=9.11%
    lat (msec) : 100=5.61%, 250=15.04%, 500=18.14%, 750=5.08%, 1000=2.98%
    lat (msec) : 2000=0.39%, >=2000=0.79%
  cpu          : usr=0.00%, sys=0.00%, ctx=544539, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=484040/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=5287: Fri Sep 26 18:39:44 2014
  read : io=2500.0MB, bw=14892 B/s, iops=3 , runt=176021418msec
    clat (usec): min=421 , max=32574K, avg=275024.72, stdev=1500874.34
     lat (usec): min=421 , max=32574K, avg=275024.94, stdev=1500874.34
    clat percentiles (usec):
     |  1.00th=[  948],  5.00th=[ 5856], 10.00th=[ 9920], 20.00th=[10048],
     | 30.00th=[10944], 40.00th=[16064], 50.00th=[30848], 60.00th=[110080],
     | 70.00th=[197632], 80.00th=[292864], 90.00th=[452608], 95.00th=[651264],
     | 99.00th=[1003520], 99.50th=[16056320], 99.90th=[16711680], 99.95th=[16711680],
     | 99.99th=[16711680]
    bw (KB/s)  : min=    0, max=  408, per=7.08%, avg=28.33, stdev=50.59
    lat (usec) : 500=0.03%, 750=0.05%, 1000=1.25%
    lat (msec) : 2=1.48%, 4=0.92%, 10=11.66%, 20=28.04%, 50=9.55%
    lat (msec) : 100=5.84%, 250=16.50%, 500=16.24%, 750=4.60%, 1000=2.75%
    lat (msec) : 2000=0.35%, >=2000=0.72%
  cpu          : usr=0.00%, sys=0.00%, ctx=649619, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=640000/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=5288: Fri Sep 26 18:39:44 2014
  read : io=2500.0MB, bw=14637 B/s, iops=3 , runt=179093756msec
    clat (usec): min=457 , max=32751K, avg=279825.14, stdev=1518373.94
     lat (usec): min=458 , max=32751K, avg=279825.37, stdev=1518373.95
    clat percentiles (usec):
     |  1.00th=[  956],  5.00th=[ 5920], 10.00th=[ 9920], 20.00th=[10048],
     | 30.00th=[10944], 40.00th=[18560], 50.00th=[31360], 60.00th=[120320],
     | 70.00th=[201728], 80.00th=[301056], 90.00th=[452608], 95.00th=[651264],
     | 99.00th=[1011712], 99.50th=[16056320], 99.90th=[16711680], 99.95th=[16711680],
     | 99.99th=[16711680]
    bw (KB/s)  : min=    0, max=  408, per=6.92%, avg=27.66, stdev=49.35
    lat (usec) : 500=0.03%, 750=0.06%, 1000=1.25%
    lat (msec) : 2=1.45%, 4=0.94%, 10=11.43%, 20=27.42%, 50=9.35%
    lat (msec) : 100=5.82%, 250=16.96%, 500=16.73%, 750=4.69%, 1000=2.79%
    lat (msec) : 2000=0.36%, >=2000=0.72%
  cpu          : usr=0.00%, sys=0.00%, ctx=653594, majf=0, minf=26
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=640000/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=5289: Fri Sep 26 18:39:44 2014
  read : io=2500.0MB, bw=14765 B/s, iops=3 , runt=177536723msec
    clat (usec): min=374 , max=32590K, avg=277391.58, stdev=1517587.53
     lat (usec): min=374 , max=32590K, avg=277391.73, stdev=1517587.53
    clat percentiles (usec):
     |  1.00th=[  948],  5.00th=[ 5920], 10.00th=[ 9920], 20.00th=[10048],
     | 30.00th=[10944], 40.00th=[16768], 50.00th=[30848], 60.00th=[111104],
     | 70.00th=[199680], 80.00th=[296960], 90.00th=[452608], 95.00th=[651264],
     | 99.00th=[1003520], 99.50th=[16056320], 99.90th=[16711680], 99.95th=[16711680],
     | 99.99th=[16711680]
    bw (KB/s)  : min=    0, max=  408, per=7.02%, avg=28.09, stdev=50.22
    lat (usec) : 500=0.03%, 750=0.05%, 1000=1.27%
    lat (msec) : 2=1.43%, 4=0.92%, 10=11.57%, 20=27.92%, 50=9.50%
    lat (msec) : 100=5.75%, 250=16.75%, 500=16.28%, 750=4.65%, 1000=2.81%
    lat (msec) : 2000=0.35%, >=2000=0.72%
  cpu          : usr=0.00%, sys=0.00%, ctx=643860, majf=0, minf=27
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=640000/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=5290: Fri Sep 26 18:39:44 2014
  read : io=2500.0MB, bw=14617 B/s, iops=3 , runt=179331294msec
    clat (usec): min=434 , max=32593K, avg=280199.79, stdev=1532841.64
     lat (usec): min=435 , max=32593K, avg=280199.98, stdev=1532841.65
    clat percentiles (usec):
     |  1.00th=[  956],  5.00th=[ 5920], 10.00th=[ 9920], 20.00th=[10048],
     | 30.00th=[10944], 40.00th=[18560], 50.00th=[31104], 60.00th=[120320],
     | 70.00th=[201728], 80.00th=[301056], 90.00th=[456704], 95.00th=[659456],
     | 99.00th=[1003520], 99.50th=[16056320], 99.90th=[16711680], 99.95th=[16711680],
     | 99.99th=[16711680]
    bw (KB/s)  : min=    0, max=  408, per=6.92%, avg=27.69, stdev=49.13
    lat (usec) : 500=0.03%, 750=0.05%, 1000=1.25%
    lat (msec) : 2=1.43%, 4=0.92%, 10=11.41%, 20=27.52%, 50=9.54%
    lat (msec) : 100=5.84%, 250=16.92%, 500=16.51%, 750=4.68%, 1000=2.82%
    lat (msec) : 2000=0.36%, >=2000=0.72%
  cpu          : usr=0.00%, sys=0.00%, ctx=645599, majf=0, minf=27
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=640000/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=5291: Fri Sep 26 18:39:44 2014
  read : io=1978.4MB, bw=13539 B/s, iops=3 , runt=153219962msec
    clat (usec): min=456 , max=32580K, avg=302520.43, stdev=1607997.78
     lat (usec): min=456 , max=32580K, avg=302520.64, stdev=1607997.78
    clat percentiles (usec):
     |  1.00th=[  956],  5.00th=[ 4960], 10.00th=[ 9792], 20.00th=[10048],
     | 30.00th=[10944], 40.00th=[17536], 50.00th=[34560], 60.00th=[127488],
     | 70.00th=[220160], 80.00th=[301056], 90.00th=[481280], 95.00th=[692224],
     | 99.00th=[1019904], 99.50th=[16187392], 99.90th=[16711680], 99.95th=[16711680],
     | 99.99th=[16711680]
    bw (KB/s)  : min=    0, max=  408, per=6.58%, avg=26.31, stdev=46.83
    lat (usec) : 500=0.05%, 750=0.05%, 1000=1.22%
    lat (msec) : 2=1.73%, 4=1.17%, 10=11.69%, 20=26.84%, 50=9.04%
    lat (msec) : 100=5.63%, 250=14.99%, 500=18.22%, 750=5.11%, 1000=3.05%
    lat (msec) : 2000=0.39%, >=2000=0.82%
  cpu          : usr=0.00%, sys=0.00%, ctx=515937, majf=0, minf=27
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=506464/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=5292: Fri Sep 26 18:39:44 2014
  read : io=2500.0MB, bw=14517 B/s, iops=3 , runt=180576289msec
    clat (usec): min=461 , max=32610K, avg=282141.08, stdev=1540052.53
     lat (usec): min=462 , max=32610K, avg=282141.58, stdev=1540052.54
    clat percentiles (usec):
     |  1.00th=[  956],  5.00th=[ 5664], 10.00th=[ 9920], 20.00th=[10048],
     | 30.00th=[10944], 40.00th=[14016], 50.00th=[30848], 60.00th=[111104],
     | 70.00th=[209920], 80.00th=[301056], 90.00th=[460800], 95.00th=[659456],
     | 99.00th=[1011712], 99.50th=[16056320], 99.90th=[16711680], 99.95th=[16711680],
     | 99.99th=[16711680]
    bw (KB/s)  : min=    0, max=  414, per=6.99%, avg=27.94, stdev=52.28
    lat (usec) : 500=0.04%, 750=0.05%, 1000=1.37%
    lat (msec) : 2=1.48%, 4=0.91%, 10=11.44%, 20=28.63%, 50=8.94%
    lat (msec) : 100=5.56%, 250=16.06%, 500=16.84%, 750=4.71%, 1000=2.85%
    lat (msec) : 2000=0.36%, >=2000=0.74%
  cpu          : usr=0.00%, sys=0.00%, ctx=697784, majf=0, minf=26
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=640000/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=5293: Fri Sep 26 18:39:44 2014
  read : io=1949.9MB, bw=13588 B/s, iops=3 , runt=150463389msec
    clat (usec): min=444 , max=32699K, avg=301425.24, stdev=1614266.25
     lat (usec): min=444 , max=32699K, avg=301425.41, stdev=1614266.26
    clat percentiles (usec):
     |  1.00th=[  964],  5.00th=[ 5024], 10.00th=[ 9792], 20.00th=[10048],
     | 30.00th=[10944], 40.00th=[17536], 50.00th=[31104], 60.00th=[120320],
     | 70.00th=[211968], 80.00th=[301056], 90.00th=[473088], 95.00th=[684032],
     | 99.00th=[1019904], 99.50th=[16187392], 99.90th=[16711680], 99.95th=[16711680],
     | 99.99th=[16711680]
    bw (KB/s)  : min=    0, max=  408, per=6.68%, avg=26.72, stdev=47.00
    lat (usec) : 500=0.04%, 750=0.06%, 1000=1.16%
    lat (msec) : 2=1.73%, 4=1.18%, 10=12.05%, 20=26.65%, 50=9.32%
    lat (msec) : 100=5.64%, 250=14.99%, 500=17.95%, 750=5.03%, 1000=2.99%
    lat (msec) : 2000=0.38%, >=2000=0.83%
  cpu          : usr=0.00%, sys=0.00%, ctx=501161, majf=0, minf=26
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=499160/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=5294: Fri Sep 26 18:39:44 2014
  read : io=2500.0MB, bw=14761 B/s, iops=3 , runt=177586156msec
    clat (usec): min=439 , max=32674K, avg=277472.44, stdev=1525836.10
     lat (usec): min=440 , max=32674K, avg=277472.62, stdev=1525836.11
    clat percentiles (usec):
     |  1.00th=[  948],  5.00th=[ 5920], 10.00th=[ 9920], 20.00th=[10048],
     | 30.00th=[10944], 40.00th=[15168], 50.00th=[30336], 60.00th=[110080],
     | 70.00th=[197632], 80.00th=[296960], 90.00th=[452608], 95.00th=[651264],
     | 99.00th=[1011712], 99.50th=[16056320], 99.90th=[16711680], 99.95th=[16711680],
     | 99.99th=[16711680]
    bw (KB/s)  : min=    0, max=  408, per=7.07%, avg=28.29, stdev=50.96
    lat (usec) : 500=0.04%, 750=0.06%, 1000=1.27%
    lat (msec) : 2=1.44%, 4=0.92%, 10=12.06%, 20=28.00%, 50=9.46%
    lat (msec) : 100=5.66%, 250=16.40%, 500=16.25%, 750=4.60%, 1000=2.79%
    lat (msec) : 2000=0.36%, >=2000=0.73%
  cpu          : usr=0.00%, sys=0.00%, ctx=643199, majf=0, minf=26
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=640000/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=5295: Fri Sep 26 18:39:44 2014
  read : io=2500.0MB, bw=14521 B/s, iops=3 , runt=180519760msec
    clat (usec): min=446 , max=32580K, avg=282056.11, stdev=1524304.17
     lat (usec): min=446 , max=32580K, avg=282056.40, stdev=1524304.18
    clat percentiles (usec):
     |  1.00th=[  956],  5.00th=[ 5792], 10.00th=[ 9920], 20.00th=[10048],
     | 30.00th=[10944], 40.00th=[17280], 50.00th=[31104], 60.00th=[121344],
     | 70.00th=[209920], 80.00th=[301056], 90.00th=[460800], 95.00th=[659456],
     | 99.00th=[1011712], 99.50th=[16056320], 99.90th=[16711680], 99.95th=[16711680],
     | 99.99th=[16711680]
    bw (KB/s)  : min=    0, max=  408, per=6.89%, avg=27.55, stdev=50.15
    lat (usec) : 500=0.04%, 750=0.05%, 1000=1.31%
    lat (msec) : 2=1.45%, 4=0.92%, 10=11.20%, 20=27.87%, 50=9.19%
    lat (msec) : 100=5.71%, 250=16.64%, 500=16.96%, 750=4.71%, 1000=2.85%
    lat (msec) : 2000=0.38%, >=2000=0.73%
  cpu          : usr=0.00%, sys=0.00%, ctx=663585, majf=0, minf=26
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=640000/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=5296: Fri Sep 26 18:39:44 2014
  read : io=2500.0MB, bw=14592 B/s, iops=3 , runt=179639086msec
    clat (usec): min=422 , max=32569K, avg=280681.40, stdev=1535233.70
     lat (usec): min=422 , max=32569K, avg=280681.58, stdev=1535233.71
    clat percentiles (usec):
     |  1.00th=[  956],  5.00th=[ 5920], 10.00th=[ 9920], 20.00th=[10048],
     | 30.00th=[10944], 40.00th=[18304], 50.00th=[31104], 60.00th=[118272],
     | 70.00th=[201728], 80.00th=[301056], 90.00th=[452608], 95.00th=[651264],
     | 99.00th=[1011712], 99.50th=[16056320], 99.90th=[16711680], 99.95th=[16711680],
     | 99.99th=[16711680]
    bw (KB/s)  : min=    0, max=  408, per=6.97%, avg=27.87, stdev=49.27
    lat (usec) : 500=0.04%, 750=0.05%, 1000=1.23%
    lat (msec) : 2=1.48%, 4=0.90%, 10=11.32%, 20=27.63%, 50=9.65%
    lat (msec) : 100=5.92%, 250=16.69%, 500=16.54%, 750=4.68%, 1000=2.79%
    lat (msec) : 2000=0.35%, >=2000=0.73%
  cpu          : usr=0.00%, sys=0.00%, ctx=645878, majf=0, minf=26
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=640000/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=5297: Fri Sep 26 18:39:44 2014
  read : io=2500.0MB, bw=14765 B/s, iops=3 , runt=177535166msec
    clat (usec): min=458 , max=32641K, avg=277391.32, stdev=1510147.00
     lat (usec): min=458 , max=32641K, avg=277391.46, stdev=1510147.00
    clat percentiles (usec):
     |  1.00th=[  948],  5.00th=[ 5920], 10.00th=[ 9920], 20.00th=[10048],
     | 30.00th=[10944], 40.00th=[16768], 50.00th=[30848], 60.00th=[111104],
     | 70.00th=[199680], 80.00th=[296960], 90.00th=[452608], 95.00th=[651264],
     | 99.00th=[1011712], 99.50th=[16056320], 99.90th=[16711680], 99.95th=[16711680],
     | 99.99th=[16711680]
    bw (KB/s)  : min=    0, max=  408, per=7.02%, avg=28.09, stdev=49.81
    lat (usec) : 500=0.04%, 750=0.05%, 1000=1.23%
    lat (msec) : 2=1.45%, 4=0.96%, 10=11.69%, 20=27.85%, 50=9.58%
    lat (msec) : 100=5.71%, 250=16.64%, 500=16.30%, 750=4.67%, 1000=2.75%
    lat (msec) : 2000=0.35%, >=2000=0.73%
  cpu          : usr=0.00%, sys=0.00%, ctx=640083, majf=0, minf=27
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=640000/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=5298: Fri Sep 26 18:39:44 2014
  read : io=1996.2MB, bw=14069 B/s, iops=3 , runt=148771618msec
    clat (usec): min=461 , max=32621K, avg=291128.47, stdev=1571335.30
     lat (usec): min=461 , max=32621K, avg=291128.63, stdev=1571335.30
    clat percentiles (usec):
     |  1.00th=[  956],  5.00th=[ 4960], 10.00th=[ 9792], 20.00th=[10048],
     | 30.00th=[10944], 40.00th=[17024], 50.00th=[31104], 60.00th=[120320],
     | 70.00th=[209920], 80.00th=[301056], 90.00th=[468992], 95.00th=[675840],
     | 99.00th=[1011712], 99.50th=[16187392], 99.90th=[16711680], 99.95th=[16711680],
     | 99.99th=[16711680]
    bw (KB/s)  : min=    0, max=  408, per=6.77%, avg=27.07, stdev=47.44
    lat (usec) : 500=0.04%, 750=0.06%, 1000=1.19%
    lat (msec) : 2=1.78%, 4=1.18%, 10=11.95%, 20=26.93%, 50=9.30%
    lat (msec) : 100=5.66%, 250=15.13%, 500=17.70%, 750=4.99%, 1000=2.95%
    lat (msec) : 2000=0.37%, >=2000=0.77%
  cpu          : usr=0.00%, sys=0.00%, ctx=513435, majf=0, minf=26
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=511004/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=5299: Fri Sep 26 18:39:44 2014
  read : io=1880.6MB, bw=13947 B/s, iops=3 , runt=141381554msec
    clat (usec): min=459 , max=32582K, avg=293669.10, stdev=1581813.45
     lat (usec): min=459 , max=32582K, avg=293669.33, stdev=1581813.46
    clat percentiles (usec):
     |  1.00th=[  956],  5.00th=[ 4896], 10.00th=[ 9664], 20.00th=[10048],
     | 30.00th=[10944], 40.00th=[15680], 50.00th=[30848], 60.00th=[111104],
     | 70.00th=[209920], 80.00th=[301056], 90.00th=[468992], 95.00th=[684032],
     | 99.00th=[1019904], 99.50th=[16187392], 99.90th=[16711680], 99.95th=[16711680],
     | 99.99th=[16711680]
    bw (KB/s)  : min=    0, max=  408, per=6.79%, avg=27.16, stdev=48.08
    lat (usec) : 500=0.04%, 750=0.06%, 1000=1.20%
    lat (msec) : 2=1.74%, 4=1.22%, 10=12.16%, 20=27.31%, 50=9.17%
    lat (msec) : 100=5.48%, 250=14.83%, 500=17.73%, 750=4.96%, 1000=2.92%
    lat (msec) : 2000=0.39%, >=2000=0.79%
  cpu          : usr=0.00%, sys=0.00%, ctx=492300, majf=0, minf=26
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=481416/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=5300: Fri Sep 26 18:39:44 2014
  read : io=2500.0MB, bw=14632 B/s, iops=3 , runt=179146940msec
    clat (usec): min=446 , max=32727K, avg=279910.31, stdev=1529479.30
     lat (usec): min=446 , max=32727K, avg=279910.70, stdev=1529479.30
    clat percentiles (usec):
     |  1.00th=[  948],  5.00th=[ 5792], 10.00th=[ 9920], 20.00th=[10048],
     | 30.00th=[10944], 40.00th=[16512], 50.00th=[31104], 60.00th=[111104],
     | 70.00th=[199680], 80.00th=[296960], 90.00th=[452608], 95.00th=[651264],
     | 99.00th=[1011712], 99.50th=[16056320], 99.90th=[16711680], 99.95th=[16711680],
     | 99.99th=[16711680]
    bw (KB/s)  : min=    0, max=  408, per=7.02%, avg=28.09, stdev=50.09
    lat (usec) : 500=0.04%, 750=0.04%, 1000=1.24%
    lat (msec) : 2=1.50%, 4=0.93%, 10=11.61%, 20=27.90%, 50=9.40%
    lat (msec) : 100=5.80%, 250=16.70%, 500=16.37%, 750=4.62%, 1000=2.74%
    lat (msec) : 2000=0.36%, >=2000=0.74%
  cpu          : usr=0.00%, sys=0.00%, ctx=678056, majf=0, minf=26
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=640000/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=5301: Fri Sep 26 18:39:44 2014
  read : io=2500.0MB, bw=14596 B/s, iops=3 , runt=179597516msec
    clat (usec): min=431 , max=32590K, avg=280611.02, stdev=1530175.84
     lat (usec): min=431 , max=32590K, avg=280611.29, stdev=1530175.85
    clat percentiles (usec):
     |  1.00th=[  948],  5.00th=[ 5856], 10.00th=[ 9920], 20.00th=[10048],
     | 30.00th=[10944], 40.00th=[17792], 50.00th=[31104], 60.00th=[114176],
     | 70.00th=[201728], 80.00th=[301056], 90.00th=[460800], 95.00th=[659456],
     | 99.00th=[1011712], 99.50th=[16056320], 99.90th=[16711680], 99.95th=[16711680],
     | 99.99th=[16711680]
    bw (KB/s)  : min=    0, max=  408, per=6.96%, avg=27.85, stdev=49.44
    lat (usec) : 500=0.04%, 750=0.05%, 1000=1.27%
    lat (msec) : 2=1.44%, 4=0.93%, 10=11.36%, 20=27.69%, 50=9.58%
    lat (msec) : 100=5.87%, 250=16.67%, 500=16.49%, 750=4.69%, 1000=2.82%
    lat (msec) : 2000=0.36%, >=2000=0.73%
  cpu          : usr=0.00%, sys=0.00%, ctx=655266, majf=0, minf=26
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=640000/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=5302: Fri Sep 26 18:39:44 2014
  read : io=2500.0MB, bw=14619 B/s, iops=3 , runt=179312864msec
    clat (usec): min=445 , max=32593K, avg=280172.70, stdev=1528392.59
     lat (usec): min=445 , max=32593K, avg=280172.90, stdev=1528392.60
    clat percentiles (usec):
     |  1.00th=[  956],  5.00th=[ 5984], 10.00th=[ 9920], 20.00th=[10048],
     | 30.00th=[10944], 40.00th=[17536], 50.00th=[31104], 60.00th=[111104],
     | 70.00th=[199680], 80.00th=[301056], 90.00th=[452608], 95.00th=[651264],
     | 99.00th=[1011712], 99.50th=[16056320], 99.90th=[16711680], 99.95th=[16711680],
     | 99.99th=[16711680]
    bw (KB/s)  : min=    0, max=  408, per=7.00%, avg=27.98, stdev=49.50
    lat (usec) : 500=0.04%, 750=0.05%, 1000=1.23%
    lat (msec) : 2=1.41%, 4=0.93%, 10=11.71%, 20=27.59%, 50=9.71%
    lat (msec) : 100=5.77%, 250=16.68%, 500=16.31%, 750=4.69%, 1000=2.78%
    lat (msec) : 2000=0.36%, >=2000=0.74%
  cpu          : usr=0.00%, sys=0.00%, ctx=648744, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=640000/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=5303: Fri Sep 26 18:39:44 2014
  read : io=2500.0MB, bw=15022 B/s, iops=3 , runt=174501361msec
    clat (usec): min=422 , max=32610K, avg=272651.91, stdev=1491153.13
     lat (usec): min=422 , max=32610K, avg=272652.05, stdev=1491153.13
    clat percentiles (usec):
     |  1.00th=[  948],  5.00th=[ 5792], 10.00th=[ 9920], 20.00th=[10048],
     | 30.00th=[10944], 40.00th=[15680], 50.00th=[30336], 60.00th=[110080],
     | 70.00th=[191488], 80.00th=[292864], 90.00th=[448512], 95.00th=[643072],
     | 99.00th=[1003520], 99.50th=[16056320], 99.90th=[16711680], 99.95th=[16711680],
     | 99.99th=[16711680]
    bw (KB/s)  : min=    0, max=  411, per=7.11%, avg=28.46, stdev=50.96
    lat (usec) : 500=0.04%, 750=0.06%, 1000=1.30%
    lat (msec) : 2=1.44%, 4=0.94%, 10=11.66%, 20=28.17%, 50=9.56%
    lat (msec) : 100=5.77%, 250=16.41%, 500=16.28%, 750=4.58%, 1000=2.72%
    lat (msec) : 2000=0.35%, >=2000=0.71%
  cpu          : usr=0.00%, sys=0.00%, ctx=640906, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=640000/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=5304: Fri Sep 26 18:39:44 2014
  read : io=2500.0MB, bw=14650 B/s, iops=3 , runt=178937809msec
    clat (usec): min=432 , max=32611K, avg=279582.01, stdev=1516082.78
     lat (usec): min=433 , max=32611K, avg=279582.29, stdev=1516082.78
    clat percentiles (usec):
     |  1.00th=[  948],  5.00th=[ 5792], 10.00th=[ 9920], 20.00th=[10048],
     | 30.00th=[10944], 40.00th=[17280], 50.00th=[31104], 60.00th=[120320],
     | 70.00th=[201728], 80.00th=[301056], 90.00th=[452608], 95.00th=[651264],
     | 99.00th=[1003520], 99.50th=[16056320], 99.90th=[16711680], 99.95th=[16711680],
     | 99.99th=[16711680]
    bw (KB/s)  : min=    0, max=  408, per=6.93%, avg=27.70, stdev=49.46
    lat (usec) : 500=0.03%, 750=0.05%, 1000=1.25%
    lat (msec) : 2=1.47%, 4=0.94%, 10=11.55%, 20=27.59%, 50=9.25%
    lat (msec) : 100=5.76%, 250=16.92%, 500=16.63%, 750=4.68%, 1000=2.79%
    lat (msec) : 2000=0.36%, >=2000=0.73%
  cpu          : usr=0.00%, sys=0.00%, ctx=660602, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=640000/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=5305: Fri Sep 26 18:39:44 2014
  read : io=1941.1MB, bw=13766 B/s, iops=3 , runt=147917721msec
    clat (usec): min=440 , max=32650K, avg=297534.19, stdev=1573852.58
     lat (usec): min=441 , max=32650K, avg=297534.36, stdev=1573852.58
    clat percentiles (usec):
     |  1.00th=[  964],  5.00th=[ 5024], 10.00th=[ 9792], 20.00th=[10048],
     | 30.00th=[10944], 40.00th=[18560], 50.00th=[39680], 60.00th=[128512],
     | 70.00th=[220160], 80.00th=[301056], 90.00th=[477184], 95.00th=[684032],
     | 99.00th=[1019904], 99.50th=[16187392], 99.90th=[16711680], 99.95th=[16711680],
     | 99.99th=[16711680]
    bw (KB/s)  : min=    0, max=  408, per=6.63%, avg=26.51, stdev=46.06
    lat (usec) : 500=0.04%, 750=0.06%, 1000=1.13%
    lat (msec) : 2=1.75%, 4=1.19%, 10=11.97%, 20=26.25%, 50=9.19%
    lat (msec) : 100=5.69%, 250=15.33%, 500=18.16%, 750=5.08%, 1000=2.97%
    lat (msec) : 2000=0.38%, >=2000=0.80%
  cpu          : usr=0.00%, sys=0.00%, ctx=500169, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=497128/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=5306: Fri Sep 26 18:39:44 2014
  read : io=2500.0MB, bw=14527 B/s, iops=3 , runt=180443876msec
    clat (usec): min=457 , max=32604K, avg=281938.92, stdev=1536098.18
     lat (usec): min=457 , max=32604K, avg=281939.26, stdev=1536098.19
    clat percentiles (usec):
     |  1.00th=[  956],  5.00th=[ 5856], 10.00th=[ 9920], 20.00th=[10048],
     | 30.00th=[10944], 40.00th=[14656], 50.00th=[30592], 60.00th=[111104],
     | 70.00th=[201728], 80.00th=[301056], 90.00th=[460800], 95.00th=[659456],
     | 99.00th=[1011712], 99.50th=[16056320], 99.90th=[16711680], 99.95th=[16711680],
     | 99.99th=[16711680]
    bw (KB/s)  : min=    0, max=  408, per=7.01%, avg=28.02, stdev=51.38
    lat (usec) : 500=0.04%, 750=0.05%, 1000=1.29%
    lat (msec) : 2=1.45%, 4=0.91%, 10=11.78%, 20=28.29%, 50=9.30%
    lat (msec) : 100=5.59%, 250=16.06%, 500=16.59%, 750=4.71%, 1000=2.83%
    lat (msec) : 2000=0.36%, >=2000=0.75%
  cpu          : usr=0.00%, sys=0.00%, ctx=679184, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=640000/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=5307: Fri Sep 26 18:39:44 2014
  read : io=1961.1MB, bw=13721 B/s, iops=3 , runt=149929864msec
    clat (usec): min=459 , max=32615K, avg=298511.87, stdev=1586482.09
     lat (usec): min=459 , max=32615K, avg=298512.18, stdev=1586482.09
    clat percentiles (usec):
     |  1.00th=[  972],  5.00th=[ 5024], 10.00th=[ 9792], 20.00th=[10048],
     | 30.00th=[10944], 40.00th=[18816], 50.00th=[40192], 60.00th=[129536],
     | 70.00th=[220160], 80.00th=[301056], 90.00th=[481280], 95.00th=[684032],
     | 99.00th=[1019904], 99.50th=[16187392], 99.90th=[16711680], 99.95th=[16711680],
     | 99.99th=[16711680]
    bw (KB/s)  : min=    0, max=  408, per=6.59%, avg=26.36, stdev=46.21
    lat (usec) : 500=0.05%, 750=0.06%, 1000=1.11%
    lat (msec) : 2=1.77%, 4=1.16%, 10=11.90%, 20=26.23%, 50=9.16%
    lat (msec) : 100=5.59%, 250=15.30%, 500=18.38%, 750=5.08%, 1000=3.01%
    lat (msec) : 2000=0.40%, >=2000=0.79%
  cpu          : usr=0.00%, sys=0.00%, ctx=518192, majf=0, minf=26
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=502244/w=0/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
   READ: io=70541MB, aggrb=400KB/s, minb=13KB/s, maxb=14KB/s, mint=141381554msec, maxt=180576289msec

sudo mkdir -p /backup/fio && sudo fio randwrite4k30.fio && sudo rm -rf /backup/fio
randwrite4k30: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=1
...
randwrite4k30: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=1
fio-2.0.13
Starting 30 processes
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
Jobs: 1 (f=1): [__________________________w___] [100.0% done] [0K/400K/0K /s] [0 /100 /0  iops] [eta 00m:15s]]
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=7571: Sun Sep 28 21:30:46 2014
  write: io=1941.1MB, bw=13765 B/s, iops=3 , runt=147922828msec
    clat (usec): min=421 , max=32641K, avg=297544.57, stdev=1513059.80
     lat (usec): min=421 , max=32641K, avg=297544.82, stdev=1513059.79
    clat percentiles (msec):
     |  1.00th=[   10],  5.00th=[   11], 10.00th=[   41], 20.00th=[   71],
     | 30.00th=[  111], 40.00th=[  153], 50.00th=[  180], 60.00th=[  190],
     | 70.00th=[  210], 80.00th=[  221], 90.00th=[  241], 95.00th=[  260],
     | 99.00th=[ 1020], 99.50th=[16319], 99.90th=[16450], 99.95th=[16450],
     | 99.99th=[16712]
    bw (KB/s)  : min=    0, max=  408, per=6.28%, avg=25.12, stdev=27.37
    lat (usec) : 500=0.03%, 750=0.04%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=3.21%, 20=3.36%, 50=7.88%
    lat (msec) : 100=12.77%, 250=66.68%, 500=4.62%, 750=0.20%, 1000=0.17%
    lat (msec) : 2000=0.19%, >=2000=0.86%
  cpu          : usr=0.00%, sys=0.00%, ctx=502824, majf=0, minf=23
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=497124/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=7572: Sun Sep 28 21:30:46 2014
  write: io=2500.0MB, bw=14588 B/s, iops=3 , runt=179695670msec
    clat (usec): min=431 , max=32482K, avg=280735.45, stdev=1527872.78
     lat (usec): min=431 , max=32482K, avg=280735.68, stdev=1527872.78
    clat percentiles (msec):
     |  1.00th=[   10],  5.00th=[   11], 10.00th=[   21], 20.00th=[   50],
     | 30.00th=[   71], 40.00th=[  111], 50.00th=[  153], 60.00th=[  180],
     | 70.00th=[  200], 80.00th=[  212], 90.00th=[  233], 95.00th=[  253],
     | 99.00th=[ 1020], 99.50th=[16319], 99.90th=[16450], 99.95th=[16450],
     | 99.99th=[16712]
    bw (KB/s)  : min=    0, max=  408, per=7.07%, avg=28.28, stdev=32.59
    lat (usec) : 500=0.03%, 750=0.05%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=3.83%, 20=5.03%, 50=13.06%
    lat (msec) : 100=16.77%, 250=55.93%, 500=3.74%, 750=0.25%, 1000=0.22%
    lat (msec) : 2000=0.23%, >=2000=0.86%
  cpu          : usr=0.00%, sys=0.00%, ctx=660949, majf=0, minf=23
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=640000/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=7573: Sun Sep 28 21:30:46 2014
  write: io=2500.0MB, bw=14569 B/s, iops=3 , runt=179922578msec
    clat (usec): min=419 , max=32575K, avg=281121.67, stdev=1524563.70
     lat (usec): min=419 , max=32575K, avg=281121.96, stdev=1524563.69
    clat percentiles (msec):
     |  1.00th=[   10],  5.00th=[   11], 10.00th=[   21], 20.00th=[   50],
     | 30.00th=[   71], 40.00th=[  111], 50.00th=[  153], 60.00th=[  180],
     | 70.00th=[  200], 80.00th=[  212], 90.00th=[  233], 95.00th=[  253],
     | 99.00th=[ 1020], 99.50th=[16319], 99.90th=[16450], 99.95th=[16450],
     | 99.99th=[16712]
    bw (KB/s)  : min=    0, max=  408, per=7.07%, avg=28.28, stdev=32.71
    lat (usec) : 500=0.03%, 750=0.06%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=3.92%, 20=5.01%, 50=12.91%
    lat (msec) : 100=16.73%, 250=56.05%, 500=3.75%, 750=0.24%, 1000=0.21%
    lat (msec) : 2000=0.22%, >=2000=0.86%
  cpu          : usr=0.00%, sys=0.00%, ctx=654013, majf=0, minf=23
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=640000/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=7574: Sun Sep 28 21:30:46 2014
  write: io=2500.0MB, bw=14533 B/s, iops=3 , runt=180376670msec
    clat (usec): min=406 , max=32525K, avg=281824.54, stdev=1530832.74
     lat (usec): min=406 , max=32525K, avg=281824.72, stdev=1530832.74
    clat percentiles (msec):
     |  1.00th=[   10],  5.00th=[   11], 10.00th=[   21], 20.00th=[   50],
     | 30.00th=[   71], 40.00th=[  111], 50.00th=[  161], 60.00th=[  180],
     | 70.00th=[  200], 80.00th=[  212], 90.00th=[  233], 95.00th=[  253],
     | 99.00th=[ 1020], 99.50th=[16319], 99.90th=[16450], 99.95th=[16450],
     | 99.99th=[16712]
    bw (KB/s)  : min=    0, max=  408, per=7.06%, avg=28.24, stdev=32.18
    lat (usec) : 500=0.03%, 750=0.05%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=3.59%, 20=4.95%, 50=13.35%
    lat (msec) : 100=16.85%, 250=55.86%, 500=3.77%, 750=0.25%, 1000=0.22%
    lat (msec) : 2000=0.22%, >=2000=0.86%
  cpu          : usr=0.00%, sys=0.00%, ctx=640041, majf=0, minf=23
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=640000/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=7575: Sun Sep 28 21:30:46 2014
  write: io=2500.0MB, bw=14667 B/s, iops=3 , runt=178719490msec
    clat (usec): min=399 , max=32751K, avg=279229.13, stdev=1523053.42
     lat (usec): min=399 , max=32751K, avg=279229.37, stdev=1523053.42
    clat percentiles (msec):
     |  1.00th=[   10],  5.00th=[   11], 10.00th=[   21], 20.00th=[   41],
     | 30.00th=[   71], 40.00th=[  102], 50.00th=[  157], 60.00th=[  180],
     | 70.00th=[  200], 80.00th=[  212], 90.00th=[  233], 95.00th=[  253],
     | 99.00th=[ 1020], 99.50th=[16319], 99.90th=[16450], 99.95th=[16450],
     | 99.99th=[16712]
    bw (KB/s)  : min=    0, max=  408, per=7.10%, avg=28.42, stdev=33.43
    lat (usec) : 500=0.03%, 750=0.06%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=4.15%, 20=5.25%, 50=13.27%
    lat (msec) : 100=16.33%, 250=55.61%, 500=3.78%, 750=0.24%, 1000=0.21%
    lat (msec) : 2000=0.22%, >=2000=0.85%
  cpu          : usr=0.00%, sys=0.00%, ctx=645191, majf=0, minf=23
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=640000/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=7576: Sun Sep 28 21:30:46 2014
  write: io=2500.0MB, bw=14588 B/s, iops=3 , runt=179688978msec
    clat (usec): min=429 , max=32901K, avg=280752.19, stdev=1524959.04
     lat (usec): min=429 , max=32901K, avg=280752.39, stdev=1524959.04
    clat percentiles (msec):
     |  1.00th=[   10],  5.00th=[   11], 10.00th=[   21], 20.00th=[   50],
     | 30.00th=[   71], 40.00th=[  111], 50.00th=[  161], 60.00th=[  180],
     | 70.00th=[  200], 80.00th=[  212], 90.00th=[  233], 95.00th=[  251],
     | 99.00th=[ 1020], 99.50th=[16319], 99.90th=[16450], 99.95th=[16450],
     | 99.99th=[16712]
    bw (KB/s)  : min=    0, max=  408, per=7.08%, avg=28.33, stdev=33.20
    lat (usec) : 500=0.03%, 750=0.05%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=4.06%, 20=5.22%, 50=12.78%
    lat (msec) : 100=16.38%, 250=56.25%, 500=3.71%, 750=0.23%, 1000=0.20%
    lat (msec) : 2000=0.21%, >=2000=0.86%
  cpu          : usr=0.00%, sys=0.00%, ctx=642070, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=640000/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=7577: Sun Sep 28 21:30:46 2014
  write: io=2500.0MB, bw=14554 B/s, iops=3 , runt=180116620msec
    clat (usec): min=421 , max=32631K, avg=281419.27, stdev=1526203.29
     lat (usec): min=422 , max=32631K, avg=281419.59, stdev=1526203.28
    clat percentiles (msec):
     |  1.00th=[   10],  5.00th=[   11], 10.00th=[   21], 20.00th=[   50],
     | 30.00th=[   71], 40.00th=[  111], 50.00th=[  153], 60.00th=[  180],
     | 70.00th=[  200], 80.00th=[  212], 90.00th=[  233], 95.00th=[  253],
     | 99.00th=[ 1020], 99.50th=[16319], 99.90th=[16450], 99.95th=[16450],
     | 99.99th=[16712]
    bw (KB/s)  : min=    0, max=  408, per=7.05%, avg=28.19, stdev=32.57
    lat (usec) : 500=0.03%, 750=0.06%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=3.99%, 20=4.86%, 50=12.44%
    lat (msec) : 100=17.33%, 250=55.99%, 500=3.72%, 750=0.25%, 1000=0.24%
    lat (msec) : 2000=0.23%, >=2000=0.86%
  cpu          : usr=0.00%, sys=0.00%, ctx=675147, majf=0, minf=23
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=640000/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=7578: Sun Sep 28 21:30:46 2014
  write: io=2500.0MB, bw=14735 B/s, iops=3 , runt=177896573msec
    clat (usec): min=423 , max=32631K, avg=277945.81, stdev=1506080.03
     lat (usec): min=424 , max=32631K, avg=277946.17, stdev=1506080.03
    clat percentiles (msec):
     |  1.00th=[   10],  5.00th=[   11], 10.00th=[   21], 20.00th=[   50],
     | 30.00th=[   71], 40.00th=[  111], 50.00th=[  161], 60.00th=[  180],
     | 70.00th=[  200], 80.00th=[  212], 90.00th=[  233], 95.00th=[  253],
     | 99.00th=[ 1020], 99.50th=[16319], 99.90th=[16450], 99.95th=[16450],
     | 99.99th=[16712]
    bw (KB/s)  : min=    0, max=  408, per=7.08%, avg=28.34, stdev=32.99
    lat (usec) : 500=0.03%, 750=0.05%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=3.91%, 20=5.18%, 50=13.28%
    lat (msec) : 100=16.25%, 250=56.01%, 500=3.78%, 750=0.24%, 1000=0.21%
    lat (msec) : 2000=0.22%, >=2000=0.84%
  cpu          : usr=0.00%, sys=0.00%, ctx=662214, majf=0, minf=22
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=640000/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=7579: Sun Sep 28 21:30:46 2014
  write: io=1890.8MB, bw=13510 B/s, iops=3 , runt=146742342msec
    clat (usec): min=434 , max=32631K, avg=303145.30, stdev=1535117.74
     lat (usec): min=434 , max=32631K, avg=303145.60, stdev=1535117.75
    clat percentiles (msec):
     |  1.00th=[   10],  5.00th=[   11], 10.00th=[   41], 20.00th=[   71],
     | 30.00th=[  121], 40.00th=[  161], 50.00th=[  180], 60.00th=[  192],
     | 70.00th=[  210], 80.00th=[  221], 90.00th=[  241], 95.00th=[  260],
     | 99.00th=[ 1029], 99.50th=[16319], 99.90th=[16450], 99.95th=[16450],
     | 99.99th=[16712]
    bw (KB/s)  : min=    0, max=  408, per=6.22%, avg=24.89, stdev=27.83
    lat (usec) : 500=0.03%, 750=0.05%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=3.61%, 20=3.49%, 50=6.67%
    lat (msec) : 100=12.87%, 250=67.15%, 500=4.72%, 750=0.20%, 1000=0.16%
    lat (msec) : 2000=0.18%, >=2000=0.88%
  cpu          : usr=0.00%, sys=0.00%, ctx=495792, majf=0, minf=23
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=484040/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=7580: Sun Sep 28 21:30:46 2014
  write: io=2500.0MB, bw=14616 B/s, iops=3 , runt=179352779msec
    clat (usec): min=430 , max=32571K, avg=280224.59, stdev=1523747.29
     lat (usec): min=430 , max=32571K, avg=280224.79, stdev=1523747.29
    clat percentiles (msec):
     |  1.00th=[   10],  5.00th=[   11], 10.00th=[   21], 20.00th=[   50],
     | 30.00th=[   71], 40.00th=[  111], 50.00th=[  161], 60.00th=[  180],
     | 70.00th=[  200], 80.00th=[  212], 90.00th=[  233], 95.00th=[  253],
     | 99.00th=[ 1020], 99.50th=[16319], 99.90th=[16450], 99.95th=[16450],
     | 99.99th=[16712]
    bw (KB/s)  : min=    0, max=  411, per=7.07%, avg=28.29, stdev=33.15
    lat (usec) : 500=0.03%, 750=0.05%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=4.15%, 20=5.21%, 50=12.55%
    lat (msec) : 100=16.63%, 250=56.05%, 500=3.81%, 750=0.24%, 1000=0.22%
    lat (msec) : 2000=0.22%, >=2000=0.85%
  cpu          : usr=0.00%, sys=0.00%, ctx=644921, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=640000/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=7581: Sun Sep 28 21:30:46 2014
  write: io=2500.0MB, bw=14553 B/s, iops=3 , runt=180120897msec
    clat (usec): min=425 , max=32533K, avg=281418.61, stdev=1527088.66
     lat (usec): min=425 , max=32533K, avg=281419.03, stdev=1527088.65
    clat percentiles (msec):
     |  1.00th=[   10],  5.00th=[   11], 10.00th=[   21], 20.00th=[   50],
     | 30.00th=[   71], 40.00th=[  101], 50.00th=[  151], 60.00th=[  180],
     | 70.00th=[  200], 80.00th=[  212], 90.00th=[  233], 95.00th=[  253],
     | 99.00th=[ 1029], 99.50th=[16319], 99.90th=[16450], 99.95th=[16450],
     | 99.99th=[16712]
    bw (KB/s)  : min=    0, max=  408, per=7.08%, avg=28.31, stdev=32.26
    lat (usec) : 500=0.03%, 750=0.05%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=3.63%, 20=4.84%, 50=13.47%
    lat (msec) : 100=17.21%, 250=55.44%, 500=3.75%, 750=0.26%, 1000=0.23%
    lat (msec) : 2000=0.23%, >=2000=0.86%
  cpu          : usr=0.00%, sys=0.00%, ctx=674791, majf=0, minf=24
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=640000/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=7582: Sun Sep 28 21:30:46 2014
  write: io=2500.0MB, bw=14521 B/s, iops=3 , runt=180515202msec
    clat (usec): min=421 , max=48331K, avg=282046.83, stdev=1539259.39
     lat (usec): min=421 , max=48331K, avg=282047.23, stdev=1539259.39
    clat percentiles (msec):
     |  1.00th=[   10],  5.00th=[   11], 10.00th=[   21], 20.00th=[   43],
     | 30.00th=[   71], 40.00th=[  102], 50.00th=[  151], 60.00th=[  180],
     | 70.00th=[  200], 80.00th=[  212], 90.00th=[  233], 95.00th=[  251],
     | 99.00th=[ 1029], 99.50th=[16319], 99.90th=[16450], 99.95th=[16450],
     | 99.99th=[16712]
    bw (KB/s)  : min=    0, max=  408, per=7.12%, avg=28.48, stdev=32.84
    lat (usec) : 500=0.03%, 750=0.05%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=3.84%, 20=5.06%, 50=13.56%
    lat (msec) : 100=16.66%, 250=55.54%, 500=3.70%, 750=0.24%, 1000=0.21%
    lat (msec) : 2000=0.23%, >=2000=0.87%
  cpu          : usr=0.00%, sys=0.00%, ctx=662605, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=640000/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=7583: Sun Sep 28 21:30:46 2014
  write: io=2500.0MB, bw=14517 B/s, iops=3 , runt=180571241msec
    clat (usec): min=423 , max=32635K, avg=282128.21, stdev=1535805.17
     lat (usec): min=423 , max=32635K, avg=282128.76, stdev=1535805.17
    clat percentiles (msec):
     |  1.00th=[   10],  5.00th=[   11], 10.00th=[   21], 20.00th=[   50],
     | 30.00th=[   71], 40.00th=[  102], 50.00th=[  151], 60.00th=[  180],
     | 70.00th=[  200], 80.00th=[  212], 90.00th=[  233], 95.00th=[  253],
     | 99.00th=[ 1029], 99.50th=[16319], 99.90th=[16450], 99.95th=[16450],
     | 99.99th=[16712]
    bw (KB/s)  : min=    0, max=  408, per=7.10%, avg=28.40, stdev=32.44
    lat (usec) : 500=0.03%, 750=0.05%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=3.66%, 20=4.98%, 50=13.44%
    lat (msec) : 100=16.99%, 250=55.57%, 500=3.73%, 750=0.23%, 1000=0.21%
    lat (msec) : 2000=0.23%, >=2000=0.87%
  cpu          : usr=0.00%, sys=0.00%, ctx=698452, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=640000/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=7584: Sun Sep 28 21:30:46 2014
  write: io=1978.4MB, bw=13805 B/s, iops=3 , runt=150269876msec
    clat (usec): min=431 , max=32821K, avg=296693.91, stdev=1506678.01
     lat (usec): min=431 , max=32821K, avg=296694.09, stdev=1506678.01
    clat percentiles (msec):
     |  1.00th=[   10],  5.00th=[   11], 10.00th=[   41], 20.00th=[   71],
     | 30.00th=[  111], 40.00th=[  161], 50.00th=[  180], 60.00th=[  190],
     | 70.00th=[  210], 80.00th=[  221], 90.00th=[  241], 95.00th=[  260],
     | 99.00th=[ 1020], 99.50th=[16319], 99.90th=[16450], 99.95th=[16450],
     | 99.99th=[16712]
    bw (KB/s)  : min=    0, max=  409, per=6.29%, avg=25.17, stdev=28.00
    lat (usec) : 500=0.03%, 750=0.04%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=3.60%, 20=3.53%, 50=7.26%
    lat (msec) : 100=13.32%, 250=66.22%, 500=4.59%, 750=0.19%, 1000=0.16%
    lat (msec) : 2000=0.19%, >=2000=0.85%
  cpu          : usr=0.00%, sys=0.00%, ctx=506488, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=506464/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=7585: Sun Sep 28 21:30:46 2014
  write: io=2500.0MB, bw=14715 B/s, iops=3 , runt=178147396msec
    clat (usec): min=431 , max=32631K, avg=278336.17, stdev=1515318.60
     lat (usec): min=431 , max=32631K, avg=278336.34, stdev=1515318.60
    clat percentiles (msec):
     |  1.00th=[   10],  5.00th=[   11], 10.00th=[   21], 20.00th=[   50],
     | 30.00th=[   71], 40.00th=[  111], 50.00th=[  153], 60.00th=[  180],
     | 70.00th=[  200], 80.00th=[  212], 90.00th=[  233], 95.00th=[  251],
     | 99.00th=[ 1020], 99.50th=[16319], 99.90th=[16450], 99.95th=[16450],
     | 99.99th=[16712]
    bw (KB/s)  : min=    0, max=  408, per=7.11%, avg=28.43, stdev=33.34
    lat (usec) : 500=0.03%, 750=0.05%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=4.05%, 20=5.11%, 50=13.18%
    lat (msec) : 100=16.49%, 250=55.87%, 500=3.70%, 750=0.24%, 1000=0.20%
    lat (msec) : 2000=0.23%, >=2000=0.85%
  cpu          : usr=0.00%, sys=0.00%, ctx=640112, majf=0, minf=24
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=640000/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=7586: Sun Sep 28 21:30:46 2014
  write: io=1949.9MB, bw=13750 B/s, iops=3 , runt=148691684msec
    clat (usec): min=431 , max=32521K, avg=297826.71, stdev=1512963.29
     lat (usec): min=431 , max=32521K, avg=297826.92, stdev=1512963.29
    clat percentiles (msec):
     |  1.00th=[   10],  5.00th=[   11], 10.00th=[   41], 20.00th=[   71],
     | 30.00th=[  111], 40.00th=[  153], 50.00th=[  180], 60.00th=[  192],
     | 70.00th=[  210], 80.00th=[  221], 90.00th=[  241], 95.00th=[  260],
     | 99.00th=[ 1020], 99.50th=[16319], 99.90th=[16450], 99.95th=[16450],
     | 99.99th=[16712]
    bw (KB/s)  : min=    0, max=  408, per=6.27%, avg=25.09, stdev=27.85
    lat (usec) : 500=0.04%, 750=0.05%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=3.57%, 20=3.51%, 50=6.99%
    lat (msec) : 100=13.24%, 250=66.61%, 500=4.61%, 750=0.19%, 1000=0.16%
    lat (msec) : 2000=0.18%, >=2000=0.86%
  cpu          : usr=0.00%, sys=0.00%, ctx=503886, majf=0, minf=24
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=499160/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=7587: Sun Sep 28 21:30:46 2014
  write: io=2500.0MB, bw=14528 B/s, iops=3 , runt=180430813msec
    clat (usec): min=409 , max=32611K, avg=281899.79, stdev=1535556.93
     lat (usec): min=410 , max=32611K, avg=281900.09, stdev=1535556.93
    clat percentiles (msec):
     |  1.00th=[   10],  5.00th=[   11], 10.00th=[   21], 20.00th=[   50],
     | 30.00th=[   71], 40.00th=[  111], 50.00th=[  153], 60.00th=[  180],
     | 70.00th=[  200], 80.00th=[  212], 90.00th=[  233], 95.00th=[  253],
     | 99.00th=[ 1020], 99.50th=[16319], 99.90th=[16450], 99.95th=[16450],
     | 99.99th=[16712]
    bw (KB/s)  : min=    0, max=  411, per=7.08%, avg=28.31, stdev=33.24
    lat (usec) : 500=0.03%, 750=0.06%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=4.18%, 20=5.14%, 50=12.47%
    lat (msec) : 100=16.94%, 250=55.85%, 500=3.78%, 750=0.25%, 1000=0.21%
    lat (msec) : 2000=0.22%, >=2000=0.86%
  cpu          : usr=0.00%, sys=0.00%, ctx=653843, majf=0, minf=24
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=640000/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=7588: Sun Sep 28 21:30:46 2014
  write: io=2500.0MB, bw=14582 B/s, iops=3 , runt=179770001msec
    clat (usec): min=419 , max=32781K, avg=280867.15, stdev=1527599.83
     lat (usec): min=419 , max=32781K, avg=280867.40, stdev=1527599.83
    clat percentiles (msec):
     |  1.00th=[   10],  5.00th=[   11], 10.00th=[   21], 20.00th=[   43],
     | 30.00th=[   71], 40.00th=[  102], 50.00th=[  153], 60.00th=[  180],
     | 70.00th=[  200], 80.00th=[  212], 90.00th=[  233], 95.00th=[  251],
     | 99.00th=[ 1020], 99.50th=[16319], 99.90th=[16450], 99.95th=[16450],
     | 99.99th=[16712]
    bw (KB/s)  : min=    0, max=  408, per=7.11%, avg=28.43, stdev=33.51
    lat (usec) : 500=0.03%, 750=0.06%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=4.28%, 20=5.34%, 50=12.77%
    lat (msec) : 100=16.50%, 250=55.74%, 500=3.72%, 750=0.25%, 1000=0.21%
    lat (msec) : 2000=0.22%, >=2000=0.86%
  cpu          : usr=0.00%, sys=0.00%, ctx=654551, majf=0, minf=24
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=640000/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=7589: Sun Sep 28 21:30:46 2014
  write: io=2500.0MB, bw=14677 B/s, iops=3 , runt=178603933msec
    clat (usec): min=420 , max=32496K, avg=279058.00, stdev=1512993.81
     lat (usec): min=420 , max=32496K, avg=279058.30, stdev=1512993.81
    clat percentiles (msec):
     |  1.00th=[   10],  5.00th=[   11], 10.00th=[   21], 20.00th=[   43],
     | 30.00th=[   71], 40.00th=[  111], 50.00th=[  161], 60.00th=[  180],
     | 70.00th=[  200], 80.00th=[  212], 90.00th=[  233], 95.00th=[  251],
     | 99.00th=[ 1020], 99.50th=[16319], 99.90th=[16450], 99.95th=[16450],
     | 99.99th=[16712]
    bw (KB/s)  : min=    0, max=  408, per=7.09%, avg=28.36, stdev=33.01
    lat (usec) : 500=0.03%, 750=0.05%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=3.96%, 20=5.17%, 50=13.35%
    lat (msec) : 100=16.20%, 250=55.97%, 500=3.74%, 750=0.24%, 1000=0.21%
    lat (msec) : 2000=0.22%, >=2000=0.85%
  cpu          : usr=0.00%, sys=0.00%, ctx=650548, majf=0, minf=24
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=640000/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=7590: Sun Sep 28 21:30:46 2014
  write: io=2500.0MB, bw=14538 B/s, iops=3 , runt=180311066msec
    clat (usec): min=425 , max=33011K, avg=281722.84, stdev=1539489.88
     lat (usec): min=425 , max=33011K, avg=281723.17, stdev=1539489.87
    clat percentiles (msec):
     |  1.00th=[   10],  5.00th=[   11], 10.00th=[   21], 20.00th=[   41],
     | 30.00th=[   71], 40.00th=[  100], 50.00th=[  151], 60.00th=[  180],
     | 70.00th=[  200], 80.00th=[  212], 90.00th=[  233], 95.00th=[  251],
     | 99.00th=[ 1029], 99.50th=[16319], 99.90th=[16450], 99.95th=[16450],
     | 99.99th=[16712]
    bw (KB/s)  : min=    0, max=  409, per=7.16%, avg=28.64, stdev=33.52
    lat (usec) : 500=0.03%, 750=0.05%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=4.14%, 20=5.20%, 50=13.33%
    lat (msec) : 100=17.03%, 250=54.99%, 500=3.67%, 750=0.25%, 1000=0.20%
    lat (msec) : 2000=0.21%, >=2000=0.87%
  cpu          : usr=0.00%, sys=0.00%, ctx=659076, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=640000/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=7591: Sun Sep 28 21:30:46 2014
  write: io=1996.2MB, bw=13792 B/s, iops=3 , runt=151756601msec
    clat (usec): min=398 , max=32621K, avg=296957.92, stdev=1513456.02
     lat (usec): min=399 , max=32621K, avg=296958.10, stdev=1513456.02
    clat percentiles (msec):
     |  1.00th=[   10],  5.00th=[   11], 10.00th=[   41], 20.00th=[   71],
     | 30.00th=[  111], 40.00th=[  153], 50.00th=[  180], 60.00th=[  190],
     | 70.00th=[  210], 80.00th=[  221], 90.00th=[  241], 95.00th=[  260],
     | 99.00th=[ 1020], 99.50th=[16319], 99.90th=[16450], 99.95th=[16450],
     | 99.99th=[16712]
    bw (KB/s)  : min=    0, max=  408, per=6.31%, avg=25.22, stdev=27.92
    lat (usec) : 500=0.03%, 750=0.05%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=3.50%, 20=3.58%, 50=7.53%
    lat (msec) : 100=12.86%, 250=66.59%, 500=4.47%, 750=0.18%, 1000=0.16%
    lat (msec) : 2000=0.18%, >=2000=0.86%
  cpu          : usr=0.00%, sys=0.00%, ctx=511027, majf=0, minf=23
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=511004/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=7592: Sun Sep 28 21:30:46 2014
  write: io=1880.6MB, bw=13690 B/s, iops=3 , runt=144036458msec
    clat (usec): min=411 , max=32626K, avg=299183.10, stdev=1518671.30
     lat (usec): min=411 , max=32626K, avg=299183.31, stdev=1518671.30
    clat percentiles (msec):
     |  1.00th=[   10],  5.00th=[   11], 10.00th=[   41], 20.00th=[   71],
     | 30.00th=[  113], 40.00th=[  161], 50.00th=[  180], 60.00th=[  192],
     | 70.00th=[  210], 80.00th=[  221], 90.00th=[  241], 95.00th=[  260],
     | 99.00th=[ 1020], 99.50th=[16319], 99.90th=[16450], 99.95th=[16450],
     | 99.99th=[16712]
    bw (KB/s)  : min=    0, max=  408, per=6.25%, avg=25.00, stdev=27.71
    lat (usec) : 500=0.03%, 750=0.05%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=3.46%, 20=3.52%, 50=7.43%
    lat (msec) : 100=12.50%, 250=66.89%, 500=4.70%, 750=0.19%, 1000=0.17%
    lat (msec) : 2000=0.19%, >=2000=0.86%
  cpu          : usr=0.00%, sys=0.00%, ctx=485199, majf=0, minf=23
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=481416/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=7593: Sun Sep 28 21:30:46 2014
  write: io=2500.0MB, bw=14540 B/s, iops=3 , runt=180289276msec
    clat (usec): min=432 , max=32561K, avg=281679.89, stdev=1533761.94
     lat (usec): min=433 , max=32561K, avg=281680.07, stdev=1533761.94
    clat percentiles (msec):
     |  1.00th=[   10],  5.00th=[   11], 10.00th=[   21], 20.00th=[   43],
     | 30.00th=[   71], 40.00th=[  100], 50.00th=[  151], 60.00th=[  180],
     | 70.00th=[  200], 80.00th=[  212], 90.00th=[  233], 95.00th=[  253],
     | 99.00th=[ 1029], 99.50th=[16319], 99.90th=[16450], 99.95th=[16450],
     | 99.99th=[16712]
    bw (KB/s)  : min=    0, max=  408, per=7.13%, avg=28.51, stdev=33.24
    lat (usec) : 500=0.02%, 750=0.05%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=3.99%, 20=5.17%, 50=13.38%
    lat (msec) : 100=16.78%, 250=55.30%, 500=3.74%, 750=0.25%, 1000=0.21%
    lat (msec) : 2000=0.23%, >=2000=0.87%
  cpu          : usr=0.00%, sys=0.00%, ctx=640069, majf=0, minf=23
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=640000/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=7594: Sun Sep 28 21:30:46 2014
  write: io=2500.0MB, bw=14722 B/s, iops=3 , runt=178059501msec
    clat (usec): min=417 , max=48400K, avg=278203.48, stdev=1513109.90
     lat (usec): min=417 , max=48400K, avg=278203.65, stdev=1513109.90
    clat percentiles (msec):
     |  1.00th=[   10],  5.00th=[   11], 10.00th=[   21], 20.00th=[   50],
     | 30.00th=[   71], 40.00th=[  111], 50.00th=[  161], 60.00th=[  180],
     | 70.00th=[  200], 80.00th=[  212], 90.00th=[  233], 95.00th=[  253],
     | 99.00th=[ 1020], 99.50th=[16319], 99.90th=[16450], 99.95th=[16450],
     | 99.99th=[16712]
    bw (KB/s)  : min=    0, max=  408, per=7.07%, avg=28.29, stdev=32.91
    lat (usec) : 500=0.02%, 750=0.05%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=3.90%, 20=5.06%, 50=13.06%
    lat (msec) : 100=16.50%, 250=56.11%, 500=3.75%, 750=0.25%, 1000=0.21%
    lat (msec) : 2000=0.23%, >=2000=0.84%
  cpu          : usr=0.00%, sys=0.00%, ctx=640519, majf=0, minf=23
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=640000/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=7595: Sun Sep 28 21:30:46 2014
  write: io=2500.0MB, bw=14597 B/s, iops=3 , runt=179584490msec
    clat (usec): min=405 , max=48621K, avg=280591.02, stdev=1528798.47
     lat (usec): min=406 , max=48621K, avg=280591.26, stdev=1528798.47
    clat percentiles (msec):
     |  1.00th=[   10],  5.00th=[   11], 10.00th=[   21], 20.00th=[   50],
     | 30.00th=[   71], 40.00th=[  100], 50.00th=[  153], 60.00th=[  180],
     | 70.00th=[  200], 80.00th=[  212], 90.00th=[  233], 95.00th=[  251],
     | 99.00th=[ 1020], 99.50th=[16319], 99.90th=[16450], 99.95th=[16450],
     | 99.99th=[16712]
    bw (KB/s)  : min=    0, max=  408, per=7.11%, avg=28.43, stdev=33.21
    lat (usec) : 500=0.03%, 750=0.06%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=4.10%, 20=5.17%, 50=13.04%
    lat (msec) : 100=16.95%, 250=55.38%, 500=3.72%, 750=0.24%, 1000=0.21%
    lat (msec) : 2000=0.23%, >=2000=0.86%
  cpu          : usr=0.00%, sys=0.00%, ctx=647449, majf=0, minf=23
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=640000/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=7596: Sun Sep 28 21:30:46 2014
  write: io=2500.0MB, bw=14521 B/s, iops=3 , runt=180517286msec
    clat (usec): min=431 , max=32621K, avg=282043.70, stdev=1534499.71
     lat (usec): min=431 , max=32621K, avg=282044.30, stdev=1534499.70
    clat percentiles (msec):
     |  1.00th=[   10],  5.00th=[   11], 10.00th=[   21], 20.00th=[   50],
     | 30.00th=[   71], 40.00th=[  111], 50.00th=[  153], 60.00th=[  180],
     | 70.00th=[  200], 80.00th=[  212], 90.00th=[  233], 95.00th=[  253],
     | 99.00th=[ 1020], 99.50th=[16319], 99.90th=[16450], 99.95th=[16450],
     | 99.99th=[16712]
    bw (KB/s)  : min=    0, max=  408, per=7.07%, avg=28.28, stdev=32.41
    lat (usec) : 500=0.03%, 750=0.05%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=3.76%, 20=5.03%, 50=13.28%
    lat (msec) : 100=16.66%, 250=55.91%, 500=3.72%, 750=0.25%, 1000=0.21%
    lat (msec) : 2000=0.22%, >=2000=0.87%
  cpu          : usr=0.00%, sys=0.00%, ctx=701006, majf=0, minf=23
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=640000/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=7597: Sun Sep 28 21:30:46 2014
  write: io=2500.0MB, bw=14517 B/s, iops=3 , runt=180577054msec
    clat (usec): min=433 , max=32641K, avg=282140.22, stdev=1531734.67
     lat (usec): min=433 , max=32641K, avg=282140.66, stdev=1531734.65
    clat percentiles (msec):
     |  1.00th=[   10],  5.00th=[   11], 10.00th=[   21], 20.00th=[   50],
     | 30.00th=[   71], 40.00th=[  111], 50.00th=[  161], 60.00th=[  180],
     | 70.00th=[  200], 80.00th=[  212], 90.00th=[  233], 95.00th=[  253],
     | 99.00th=[ 1020], 99.50th=[16319], 99.90th=[16450], 99.95th=[16450],
     | 99.99th=[16712]
    bw (KB/s)  : min=    0, max=  418, per=7.06%, avg=28.23, stdev=32.49
    lat (usec) : 500=0.03%, 750=0.05%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=3.79%, 20=4.86%, 50=13.34%
    lat (msec) : 100=16.73%, 250=55.84%, 500=3.75%, 750=0.27%, 1000=0.24%
    lat (msec) : 2000=0.23%, >=2000=0.86%
  cpu          : usr=0.00%, sys=0.00%, ctx=684833, majf=0, minf=23
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=640000/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=7598: Sun Sep 28 21:30:46 2014
  write: io=1941.1MB, bw=13853 B/s, iops=3 , runt=146981363msec
    clat (usec): min=411 , max=32611K, avg=295621.26, stdev=1496034.27
     lat (usec): min=411 , max=32611K, avg=295621.45, stdev=1496034.27
    clat percentiles (msec):
     |  1.00th=[   10],  5.00th=[   11], 10.00th=[   41], 20.00th=[   71],
     | 30.00th=[  113], 40.00th=[  161], 50.00th=[  180], 60.00th=[  192],
     | 70.00th=[  210], 80.00th=[  221], 90.00th=[  241], 95.00th=[  260],
     | 99.00th=[ 1012], 99.50th=[16319], 99.90th=[16450], 99.95th=[16450],
     | 99.99th=[16712]
    bw (KB/s)  : min=    0, max=  408, per=6.24%, avg=24.96, stdev=27.68
    lat (usec) : 500=0.03%, 750=0.05%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=3.55%, 20=3.44%, 50=7.09%
    lat (msec) : 100=12.82%, 250=66.96%, 500=4.67%, 750=0.21%, 1000=0.17%
    lat (msec) : 2000=0.18%, >=2000=0.84%
  cpu          : usr=0.00%, sys=0.00%, ctx=498120, majf=0, minf=23
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=497128/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=7599: Sun Sep 28 21:30:46 2014
  write: io=2500.0MB, bw=14532 B/s, iops=3 , runt=180387952msec
    clat (usec): min=403 , max=48561K, avg=281846.95, stdev=1536658.18
     lat (usec): min=404 , max=48561K, avg=281847.25, stdev=1536658.18
    clat percentiles (msec):
     |  1.00th=[   10],  5.00th=[   11], 10.00th=[   21], 20.00th=[   50],
     | 30.00th=[   71], 40.00th=[  111], 50.00th=[  151], 60.00th=[  180],
     | 70.00th=[  200], 80.00th=[  212], 90.00th=[  233], 95.00th=[  253],
     | 99.00th=[ 1029], 99.50th=[16319], 99.90th=[16450], 99.95th=[16450],
     | 99.99th=[16712]
    bw (KB/s)  : min=    0, max=  408, per=7.08%, avg=28.31, stdev=32.23
    lat (usec) : 500=0.03%, 750=0.05%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=3.71%, 20=4.88%, 50=13.05%
    lat (msec) : 100=17.22%, 250=55.75%, 500=3.73%, 750=0.25%, 1000=0.23%
    lat (msec) : 2000=0.23%, >=2000=0.86%
  cpu          : usr=0.00%, sys=0.00%, ctx=665133, majf=0, minf=22
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=640000/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=7600: Sun Sep 28 21:30:46 2014
  write: io=1961.1MB, bw=13669 B/s, iops=3 , runt=150492765msec
    clat (usec): min=431 , max=32523K, avg=299624.89, stdev=1513060.90
     lat (usec): min=431 , max=32523K, avg=299625.07, stdev=1513060.90
    clat percentiles (msec):
     |  1.00th=[   10],  5.00th=[   11], 10.00th=[   41], 20.00th=[   71],
     | 30.00th=[  121], 40.00th=[  161], 50.00th=[  180], 60.00th=[  192],
     | 70.00th=[  210], 80.00th=[  221], 90.00th=[  241], 95.00th=[  260],
     | 99.00th=[ 1020], 99.50th=[16319], 99.90th=[16450], 99.95th=[16450],
     | 99.99th=[16712]
    bw (KB/s)  : min=    0, max=  408, per=6.22%, avg=24.87, stdev=27.31
    lat (usec) : 500=0.03%, 750=0.04%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=3.50%, 20=3.34%, 50=6.80%
    lat (msec) : 100=12.73%, 250=67.54%, 500=4.63%, 750=0.19%, 1000=0.16%
    lat (msec) : 2000=0.18%, >=2000=0.86%
  cpu          : usr=0.00%, sys=0.00%, ctx=502896, majf=0, minf=24
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=502244/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
  WRITE: io=70541MB, aggrb=400KB/s, minb=13KB/s, maxb=14KB/s, mint=144036458msec, maxt=180577054msec

sudo mkdir -p /backup/fio && sudo fio seqread.fio       && sudo rm -rf /backup/fio
seqread: (g=0): rw=read, bs=1M-1M/1M-1M/1M-1M, ioengine=sync, iodepth=1
fio-2.0.13
Starting 1 process
seqread: Laying out IO file(s) (1 file(s) / 10240MB)
Jobs: 1 (f=1): [R] [100.0% done] [6144K/0K/0K /s] [6 /0 /0  iops] [eta 00m:00s]
seqread: (groupid=0, jobs=1): err= 0: pid=9825: Mon Sep 29 00:42:00 2014
  read : io=51200MB, bw=6401.1KB/s, iops=6 , runt=8190605msec
    clat (msec): min=9 , max=623 , avg=159.97, stdev= 4.38
     lat (msec): min=9 , max=623 , avg=159.97, stdev= 4.38
    clat percentiles (msec):
     |  1.00th=[  151],  5.00th=[  151], 10.00th=[  155], 20.00th=[  161],
     | 30.00th=[  161], 40.00th=[  161], 50.00th=[  161], 60.00th=[  161],
     | 70.00th=[  161], 80.00th=[  161], 90.00th=[  161], 95.00th=[  165],
     | 99.00th=[  167], 99.50th=[  167], 99.90th=[  167], 99.95th=[  167],
     | 99.99th=[  172]
    bw (KB/s)  : min= 1643, max=15944, per=100.00%, avg=6408.77, stdev=108.94
    lat (msec) : 10=0.01%, 20=0.01%, 50=0.01%, 250=99.98%, 750=0.01%
  cpu          : usr=0.00%, sys=0.03%, ctx=51251, majf=0, minf=281
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=51200/w=0/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
   READ: io=51200MB, aggrb=6401KB/s, minb=6401KB/s, maxb=6401KB/s, mint=8190605msec, maxt=8190605msec

sudo mkdir -p /backup/fio && sudo fio seqwrite.fio      && sudo rm -rf /backup/fio
seqwrite: (g=0): rw=write, bs=1M-1M/1M-1M/1M-1M, ioengine=sync, iodepth=1
fio-2.0.13
Starting 1 process
seqwrite: Laying out IO file(s) (1 file(s) / 10240MB)
Jobs: 1 (f=1): [W] [100.0% done] [0K/6144K/0K /s] [0 /6 /0  iops] [eta 00m:00s]
seqwrite: (groupid=0, jobs=1): err= 0: pid=9954: Mon Sep 29 03:25:53 2014
  write: io=51200MB, bw=6400.8KB/s, iops=6 , runt=8191118msec
    clat (msec): min=9 , max=178 , avg=159.95, stdev= 3.11
     lat (msec): min=9 , max=178 , avg=159.98, stdev= 3.11
    clat percentiles (msec):
     |  1.00th=[  151],  5.00th=[  155], 10.00th=[  157], 20.00th=[  159],
     | 30.00th=[  159], 40.00th=[  159], 50.00th=[  161], 60.00th=[  161],
     | 70.00th=[  161], 80.00th=[  163], 90.00th=[  163], 95.00th=[  163],
     | 99.00th=[  169], 99.50th=[  169], 99.90th=[  174], 99.95th=[  174],
     | 99.99th=[  176]
    bw (KB/s)  : min= 5699, max=10205, per=100.00%, avg=6408.12, stdev=48.08
    lat (msec) : 10=0.01%, 50=0.01%, 250=99.99%
  cpu          : usr=0.02%, sys=0.03%, ctx=51428, majf=0, minf=23
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=51200/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
  WRITE: io=51200MB, aggrb=6400KB/s, minb=6400KB/s, maxb=6400KB/s, mint=8191118msec, maxt=8191118msec

sudo mkdir -p /backup/fio && sudo fio randread4k.fio    && sudo rm -rf /backup/fio
randread4k: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=1
fio-2.0.13
Starting 1 process
randread4k: Laying out IO file(s) (1 file(s) / 1024MB)
Jobs: 1 (f=1): [r] [100.0% done] [396K/0K/0K /s] [99 /0 /0  iops] [eta 00m:00s]
randread4k: (groupid=0, jobs=1): err= 0: pid=10300: Mon Sep 29 07:09:53 2014
  read : io=5120.0MB, bw=409644 B/s, iops=100 , runt=13105771msec
    clat (usec): min=361 , max=54895 , avg=9994.66, stdev=2315.67
     lat (usec): min=361 , max=54896 , avg=9995.67, stdev=2315.69
    clat percentiles (usec):
     |  1.00th=[  844],  5.00th=[ 9664], 10.00th=[ 9920], 20.00th=[ 9920],
     | 30.00th=[10048], 40.00th=[10048], 50.00th=[10048], 60.00th=[10048],
     | 70.00th=[10048], 80.00th=[10176], 90.00th=[10944], 95.00th=[11072],
     | 99.00th=[20352], 99.50th=[21376], 99.90th=[23168], 99.95th=[24192],
     | 99.99th=[30080]
    bw (KB/s)  : min=  357, max=  607, per=100.00%, avg=400.25, stdev= 8.10
    lat (usec) : 500=0.77%, 750=0.04%, 1000=0.29%
    lat (msec) : 2=2.69%, 4=0.01%, 10=49.06%, 20=45.97%, 50=1.17%
    lat (msec) : 100=0.01%
  cpu          : usr=0.08%, sys=0.06%, ctx=1500454, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=1310720/w=0/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
   READ: io=5120.0MB, aggrb=400KB/s, minb=400KB/s, maxb=400KB/s, mint=13105771msec, maxt=13105771msec

sudo mkdir -p /backup/fio && sudo fio randwrite4k.fio   && sudo rm -rf /backup/fio
randwrite4k: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=1
fio-2.0.13
Starting 1 process
randwrite4k: Laying out IO file(s) (1 file(s) / 1024MB)
Jobs: 1 (f=1): [w] [100.0% done] [0K/400K/0K /s] [0 /100 /0  iops] [eta 00m:00s]
randwrite4k: (groupid=0, jobs=1): err= 0: pid=10439: Mon Sep 29 10:51:06 2014
  write: io=5120.0MB, bw=409647 B/s, iops=100 , runt=13105669msec
    clat (usec): min=415 , max=527948 , avg=9995.70, stdev=1620.36
     lat (usec): min=415 , max=527948 , avg=9996.30, stdev=1620.28
    clat percentiles (usec):
     |  1.00th=[  494],  5.00th=[ 9920], 10.00th=[ 9920], 20.00th=[ 9920],
     | 30.00th=[ 9920], 40.00th=[10048], 50.00th=[10048], 60.00th=[10048],
     | 70.00th=[10048], 80.00th=[10048], 90.00th=[10048], 95.00th=[10048],
     | 99.00th=[18304], 99.50th=[21120], 99.90th=[21376], 99.95th=[21376],
     | 99.99th=[22144]
    bw (KB/s)  : min=  152, max=  866, per=100.00%, avg=400.29, stdev= 9.77
    lat (usec) : 500=1.08%, 750=0.18%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=55.14%, 20=42.67%, 50=0.90%
    lat (msec) : 100=0.01%, 750=0.01%
  cpu          : usr=0.08%, sys=0.06%, ctx=1589512, majf=0, minf=23
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=1310720/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
  WRITE: io=5120.0MB, aggrb=400KB/s, minb=400KB/s, maxb=400KB/s, mint=13105669msec, maxt=13105669msec

sudo umount /backup


仮想サーバで構築された冗長化 NFS サーバに対する測定をします。


sudo mount -t nfs 10.110.88.58:/backup /backup

cat /proc/mounts | grep backup
10.110.88.58:/backup /backup nfs4 rw,relatime,vers=4,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.110.88.59,minorversion=0,local_lock=none,addr=10.110.88.58 0 0

df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda2             917G  3.3G  867G   1% /
tmpfs                 3.9G     0  3.9G   0% /dev/shm
/dev/sda1             248M  101M  135M  43% /boot
10.110.88.58:/backup   20G  708M   18G   4% /backup

sudo mkdir -p /backup/fio && sudo fio randread4k30.fio  && sudo rm -rf /backup/fio
randread4k30: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=1
...
randread4k30: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=1
fio-2.0.13
Starting 30 processes
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
Jobs: 1 (f=1): [__r___________________________] [99.9% done] [6848K/0K/0K /s] [1712 /0 /0  iops] [eta 00m:02s] s]
randread4k30: (groupid=0, jobs=1): err= 0: pid=21790: Tue Sep 23 20:43:00 2014
  read : io=1941.1MB, bw=986092 B/s, iops=240 , runt=2064938msec
    clat (usec): min=365 , max=59430 , avg=4152.54, stdev=2482.65
     lat (usec): min=365 , max=59430 , avg=4152.61, stdev=2482.66
    clat percentiles (usec):
     |  1.00th=[ 1112],  5.00th=[ 1816], 10.00th=[ 2192], 20.00th=[ 2608],
     | 30.00th=[ 2928], 40.00th=[ 3184], 50.00th=[ 3472], 60.00th=[ 3792],
     | 70.00th=[ 4192], 80.00th=[ 4960], 90.00th=[ 7136], 95.00th=[ 9664],
     | 99.00th=[13504], 99.50th=[15168], 99.90th=[20864], 99.95th=[23168],
     | 99.99th=[28288]
    bw (KB/s)  : min=  311, max= 2280, per=3.18%, avg=963.79, stdev=322.64
    lat (usec) : 500=0.01%, 750=0.29%, 1000=0.36%
    lat (msec) : 2=6.38%, 4=58.46%, 10=30.04%, 20=4.33%, 50=0.13%
    lat (msec) : 100=0.01%
  cpu          : usr=0.07%, sys=0.10%, ctx=503313, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=497124/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=21791: Tue Sep 23 20:43:00 2014
  read : io=2500.0MB, bw=1075.7KB/s, iops=268 , runt=2380046msec
    clat (usec): min=208 , max=44101 , avg=3717.54, stdev=2391.12
     lat (usec): min=208 , max=44101 , avg=3717.61, stdev=2391.13
    clat percentiles (usec):
     |  1.00th=[  374],  5.00th=[ 1256], 10.00th=[ 1656], 20.00th=[ 2192],
     | 30.00th=[ 2576], 40.00th=[ 2928], 50.00th=[ 3216], 60.00th=[ 3536],
     | 70.00th=[ 3920], 80.00th=[ 4576], 90.00th=[ 6176], 95.00th=[ 8768],
     | 99.00th=[12864], 99.50th=[14528], 99.90th=[19840], 99.95th=[22656],
     | 99.99th=[27520]
    bw (KB/s)  : min=  309, max= 5552, per=3.55%, avg=1076.78, stdev=482.75
    lat (usec) : 250=0.03%, 500=1.94%, 750=1.12%, 1000=0.77%
    lat (msec) : 2=11.35%, 4=56.34%, 10=25.03%, 20=3.34%, 50=0.09%
  cpu          : usr=0.09%, sys=0.12%, ctx=672867, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=640000/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=21792: Tue Sep 23 20:43:00 2014
  read : io=2500.0MB, bw=1073.6KB/s, iops=268 , runt=2384725msec
    clat (usec): min=221 , max=48568 , avg=3724.84, stdev=2427.93
     lat (usec): min=221 , max=48568 , avg=3724.91, stdev=2427.93
    clat percentiles (usec):
     |  1.00th=[  370],  5.00th=[ 1160], 10.00th=[ 1640], 20.00th=[ 2192],
     | 30.00th=[ 2576], 40.00th=[ 2896], 50.00th=[ 3184], 60.00th=[ 3504],
     | 70.00th=[ 3920], 80.00th=[ 4576], 90.00th=[ 6304], 95.00th=[ 9024],
     | 99.00th=[12864], 99.50th=[14528], 99.90th=[19840], 99.95th=[22400],
     | 99.99th=[27264]
    bw (KB/s)  : min=  284, max= 5844, per=3.55%, avg=1073.95, stdev=495.39
    lat (usec) : 250=0.04%, 500=2.13%, 750=1.13%, 1000=0.79%
    lat (msec) : 2=11.55%, 4=55.94%, 10=24.87%, 20=3.46%, 50=0.10%
  cpu          : usr=0.08%, sys=0.11%, ctx=659152, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=640000/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=21793: Tue Sep 23 20:43:00 2014
  read : io=2500.0MB, bw=1074.3KB/s, iops=268 , runt=2383547msec
    clat (usec): min=210 , max=46209 , avg=3723.06, stdev=2407.53
     lat (usec): min=210 , max=46209 , avg=3723.13, stdev=2407.53
    clat percentiles (usec):
     |  1.00th=[  370],  5.00th=[ 1224], 10.00th=[ 1656], 20.00th=[ 2192],
     | 30.00th=[ 2576], 40.00th=[ 2928], 50.00th=[ 3216], 60.00th=[ 3536],
     | 70.00th=[ 3920], 80.00th=[ 4576], 90.00th=[ 6240], 95.00th=[ 8896],
     | 99.00th=[12864], 99.50th=[14528], 99.90th=[19840], 99.95th=[22400],
     | 99.99th=[27520]
    bw (KB/s)  : min=  308, max= 5280, per=3.55%, avg=1074.56, stdev=486.84
    lat (usec) : 250=0.05%, 500=2.06%, 750=1.10%, 1000=0.74%
    lat (msec) : 2=11.39%, 4=56.13%, 10=25.06%, 20=3.37%, 50=0.10%
  cpu          : usr=0.08%, sys=0.12%, ctx=667915, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=640000/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=21794: Tue Sep 23 20:43:00 2014
  read : io=2500.0MB, bw=1074.7KB/s, iops=268 , runt=2382218msec
    clat (usec): min=208 , max=61901 , avg=3720.87, stdev=2410.51
     lat (usec): min=208 , max=61901 , avg=3720.94, stdev=2410.52
    clat percentiles (usec):
     |  1.00th=[  370],  5.00th=[ 1224], 10.00th=[ 1656], 20.00th=[ 2192],
     | 30.00th=[ 2576], 40.00th=[ 2928], 50.00th=[ 3216], 60.00th=[ 3504],
     | 70.00th=[ 3920], 80.00th=[ 4576], 90.00th=[ 6240], 95.00th=[ 8896],
     | 99.00th=[12864], 99.50th=[14528], 99.90th=[19840], 99.95th=[22400],
     | 99.99th=[28288]
    bw (KB/s)  : min=  260, max= 5668, per=3.55%, avg=1075.24, stdev=486.90
    lat (usec) : 250=0.04%, 500=2.03%, 750=1.10%, 1000=0.77%
    lat (msec) : 2=11.45%, 4=56.17%, 10=24.98%, 20=3.36%, 50=0.10%
    lat (msec) : 100=0.01%
  cpu          : usr=0.08%, sys=0.11%, ctx=662688, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=640000/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=21795: Tue Sep 23 20:43:00 2014
  read : io=2500.0MB, bw=1082.9KB/s, iops=270 , runt=2364169msec
    clat (usec): min=197 , max=39010 , avg=3692.76, stdev=2294.51
     lat (usec): min=197 , max=39010 , avg=3692.83, stdev=2294.51
    clat percentiles (usec):
     |  1.00th=[  422],  5.00th=[ 1304], 10.00th=[ 1752], 20.00th=[ 2224],
     | 30.00th=[ 2608], 40.00th=[ 2928], 50.00th=[ 3216], 60.00th=[ 3504],
     | 70.00th=[ 3888], 80.00th=[ 4512], 90.00th=[ 5984], 95.00th=[ 8384],
     | 99.00th=[12736], 99.50th=[14272], 99.90th=[19328], 99.95th=[21888],
     | 99.99th=[26752]
    bw (KB/s)  : min=  299, max= 5456, per=3.58%, avg=1083.63, stdev=433.21
    lat (usec) : 250=0.03%, 500=1.15%, 750=1.04%, 1000=0.75%
    lat (msec) : 2=11.27%, 4=57.75%, 10=24.91%, 20=3.02%, 50=0.09%
  cpu          : usr=0.08%, sys=0.11%, ctx=650123, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=640000/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=21796: Tue Sep 23 20:43:00 2014
  read : io=2500.0MB, bw=1099.5KB/s, iops=274 , runt=2328503msec
    clat (usec): min=346 , max=79190 , avg=3637.06, stdev=2160.81
     lat (usec): min=346 , max=79190 , avg=3637.13, stdev=2160.81
    clat percentiles (usec):
     |  1.00th=[  900],  5.00th=[ 1464], 10.00th=[ 1816], 20.00th=[ 2256],
     | 30.00th=[ 2576], 40.00th=[ 2896], 50.00th=[ 3184], 60.00th=[ 3472],
     | 70.00th=[ 3856], 80.00th=[ 4448], 90.00th=[ 5664], 95.00th=[ 7712],
     | 99.00th=[12480], 99.50th=[14016], 99.90th=[19072], 99.95th=[21632],
     | 99.99th=[26752]
    bw (KB/s)  : min=  307, max= 2096, per=3.63%, avg=1100.53, stdev=352.25
    lat (usec) : 500=0.01%, 750=0.72%, 1000=0.87%
    lat (msec) : 2=12.01%, 4=59.24%, 10=24.50%, 20=2.57%, 50=0.08%
    lat (msec) : 100=0.01%
  cpu          : usr=0.08%, sys=0.12%, ctx=651716, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=640000/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=21797: Tue Sep 23 20:43:00 2014
  read : io=2500.0MB, bw=1097.8KB/s, iops=274 , runt=2332142msec
    clat (usec): min=345 , max=48426 , avg=3642.68, stdev=2163.73
     lat (usec): min=345 , max=48426 , avg=3642.75, stdev=2163.73
    clat percentiles (usec):
     |  1.00th=[  900],  5.00th=[ 1464], 10.00th=[ 1816], 20.00th=[ 2256],
     | 30.00th=[ 2576], 40.00th=[ 2928], 50.00th=[ 3184], 60.00th=[ 3504],
     | 70.00th=[ 3888], 80.00th=[ 4448], 90.00th=[ 5664], 95.00th=[ 7776],
     | 99.00th=[12480], 99.50th=[14016], 99.90th=[18816], 99.95th=[21632],
     | 99.99th=[27008]
    bw (KB/s)  : min=  291, max= 2112, per=3.63%, avg=1098.75, stdev=351.85
    lat (usec) : 500=0.01%, 750=0.78%, 1000=0.88%
    lat (msec) : 2=11.96%, 4=59.06%, 10=24.64%, 20=2.60%, 50=0.07%
  cpu          : usr=0.08%, sys=0.12%, ctx=654000, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=640000/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=21798: Tue Sep 23 20:43:00 2014
  read : io=1890.8MB, bw=993.79KB/s, iops=248 , runt=1948309msec
    clat (usec): min=695 , max=52381 , avg=4023.83, stdev=2284.53
     lat (usec): min=695 , max=52381 , avg=4023.90, stdev=2284.54
    clat percentiles (usec):
     |  1.00th=[ 1432],  5.00th=[ 1848], 10.00th=[ 2192], 20.00th=[ 2608],
     | 30.00th=[ 2928], 40.00th=[ 3152], 50.00th=[ 3440], 60.00th=[ 3728],
     | 70.00th=[ 4128], 80.00th=[ 4768], 90.00th=[ 6368], 95.00th=[ 8896],
     | 99.00th=[13120], 99.50th=[14656], 99.90th=[20096], 99.95th=[22656],
     | 99.99th=[27776]
    bw (KB/s)  : min=  296, max= 1540, per=3.28%, avg=994.81, stdev=288.11
    lat (usec) : 750=0.01%, 1000=0.04%
    lat (msec) : 2=6.13%, 4=60.87%, 10=29.40%, 20=3.46%, 50=0.10%
    lat (msec) : 100=0.01%
  cpu          : usr=0.07%, sys=0.11%, ctx=495080, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=484040/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=21799: Tue Sep 23 20:43:00 2014
  read : io=2500.0MB, bw=1097.2KB/s, iops=274 , runt=2333299msec
    clat (usec): min=344 , max=43649 , avg=3644.56, stdev=2178.39
     lat (usec): min=344 , max=43649 , avg=3644.63, stdev=2178.40
    clat percentiles (usec):
     |  1.00th=[  900],  5.00th=[ 1464], 10.00th=[ 1816], 20.00th=[ 2224],
     | 30.00th=[ 2576], 40.00th=[ 2896], 50.00th=[ 3184], 60.00th=[ 3504],
     | 70.00th=[ 3856], 80.00th=[ 4448], 90.00th=[ 5728], 95.00th=[ 7840],
     | 99.00th=[12480], 99.50th=[14016], 99.90th=[19072], 99.95th=[21888],
     | 99.99th=[26752]
    bw (KB/s)  : min=  300, max= 2182, per=3.63%, avg=1098.32, stdev=353.40
    lat (usec) : 500=0.01%, 750=0.78%, 1000=0.87%
    lat (msec) : 2=12.07%, 4=59.06%, 10=24.48%, 20=2.65%, 50=0.08%
  cpu          : usr=0.08%, sys=0.11%, ctx=646172, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=640000/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=21800: Tue Sep 23 20:43:00 2014
  read : io=2500.0MB, bw=1091.8KB/s, iops=272 , runt=2346316msec
    clat (usec): min=336 , max=76378 , avg=3664.86, stdev=2207.62
     lat (usec): min=337 , max=76378 , avg=3664.94, stdev=2207.62
    clat percentiles (usec):
     |  1.00th=[  748],  5.00th=[ 1448], 10.00th=[ 1816], 20.00th=[ 2256],
     | 30.00th=[ 2608], 40.00th=[ 2928], 50.00th=[ 3184], 60.00th=[ 3504],
     | 70.00th=[ 3888], 80.00th=[ 4448], 90.00th=[ 5792], 95.00th=[ 8032],
     | 99.00th=[12480], 99.50th=[14016], 99.90th=[19072], 99.95th=[21632],
     | 99.99th=[27008]
    bw (KB/s)  : min=  313, max= 2472, per=3.61%, avg=1092.18, stdev=355.14
    lat (usec) : 500=0.03%, 750=0.99%, 1000=0.91%
    lat (msec) : 2=11.63%, 4=58.91%, 10=24.71%, 20=2.74%, 50=0.08%
    lat (msec) : 100=0.01%
  cpu          : usr=0.09%, sys=0.11%, ctx=652252, majf=0, minf=26
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=640000/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=21801: Tue Sep 23 20:43:00 2014
  read : io=2500.0MB, bw=1094.2KB/s, iops=273 , runt=2339694msec
    clat (usec): min=350 , max=45187 , avg=3654.56, stdev=2189.15
     lat (usec): min=350 , max=45188 , avg=3654.63, stdev=2189.15
    clat percentiles (usec):
     |  1.00th=[  876],  5.00th=[ 1464], 10.00th=[ 1816], 20.00th=[ 2256],
     | 30.00th=[ 2608], 40.00th=[ 2928], 50.00th=[ 3184], 60.00th=[ 3504],
     | 70.00th=[ 3888], 80.00th=[ 4448], 90.00th=[ 5728], 95.00th=[ 7904],
     | 99.00th=[12480], 99.50th=[14144], 99.90th=[19072], 99.95th=[21888],
     | 99.99th=[27008]
    bw (KB/s)  : min=  308, max= 2208, per=3.62%, avg=1095.43, stdev=353.56
    lat (usec) : 500=0.01%, 750=0.87%, 1000=0.90%
    lat (msec) : 2=11.77%, 4=59.03%, 10=24.64%, 20=2.70%, 50=0.08%
  cpu          : usr=0.08%, sys=0.11%, ctx=646692, majf=0, minf=27
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=640000/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=21802: Tue Sep 23 20:43:00 2014
  read : io=2500.0MB, bw=1092.2KB/s, iops=273 , runt=2344048msec
    clat (usec): min=336 , max=61464 , avg=3661.31, stdev=2197.58
     lat (usec): min=337 , max=61464 , avg=3661.38, stdev=2197.59
    clat percentiles (usec):
     |  1.00th=[  748],  5.00th=[ 1448], 10.00th=[ 1816], 20.00th=[ 2256],
     | 30.00th=[ 2608], 40.00th=[ 2928], 50.00th=[ 3184], 60.00th=[ 3504],
     | 70.00th=[ 3888], 80.00th=[ 4448], 90.00th=[ 5728], 95.00th=[ 7968],
     | 99.00th=[12608], 99.50th=[14144], 99.90th=[19072], 99.95th=[21632],
     | 99.99th=[26752]
    bw (KB/s)  : min=  310, max= 2286, per=3.61%, avg=1093.17, stdev=353.12
    lat (usec) : 500=0.02%, 750=0.98%, 1000=0.89%
    lat (msec) : 2=11.61%, 4=59.02%, 10=24.68%, 20=2.71%, 50=0.08%
    lat (msec) : 100=0.01%
  cpu          : usr=0.08%, sys=0.12%, ctx=650797, majf=0, minf=27
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=640000/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=21803: Tue Sep 23 20:43:00 2014
  read : io=1978.4MB, bw=1003.4KB/s, iops=250 , runt=2019172msec
    clat (usec): min=533 , max=58367 , avg=3985.42, stdev=2295.31
     lat (usec): min=533 , max=58367 , avg=3985.50, stdev=2295.32
    clat percentiles (usec):
     |  1.00th=[ 1256],  5.00th=[ 1800], 10.00th=[ 2192], 20.00th=[ 2576],
     | 30.00th=[ 2896], 40.00th=[ 3120], 50.00th=[ 3408], 60.00th=[ 3696],
     | 70.00th=[ 4128], 80.00th=[ 4768], 90.00th=[ 6304], 95.00th=[ 8896],
     | 99.00th=[12992], 99.50th=[14656], 99.90th=[20096], 99.95th=[22656],
     | 99.99th=[27776]
    bw (KB/s)  : min=  289, max= 1668, per=3.32%, avg=1004.29, stdev=291.48
    lat (usec) : 750=0.07%, 1000=0.23%
    lat (msec) : 2=7.10%, 4=60.20%, 10=28.83%, 20=3.45%, 50=0.10%
    lat (msec) : 100=0.01%
  cpu          : usr=0.07%, sys=0.10%, ctx=511706, majf=0, minf=27
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=506464/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=21804: Tue Sep 23 20:43:00 2014
  read : io=2500.0MB, bw=1090.7KB/s, iops=272 , runt=2347221msec
    clat (usec): min=271 , max=54295 , avg=3666.32, stdev=2209.13
     lat (usec): min=271 , max=54295 , avg=3666.39, stdev=2209.13
    clat percentiles (usec):
     |  1.00th=[  740],  5.00th=[ 1448], 10.00th=[ 1816], 20.00th=[ 2256],
     | 30.00th=[ 2608], 40.00th=[ 2928], 50.00th=[ 3184], 60.00th=[ 3504],
     | 70.00th=[ 3888], 80.00th=[ 4448], 90.00th=[ 5792], 95.00th=[ 8032],
     | 99.00th=[12608], 99.50th=[14144], 99.90th=[19072], 99.95th=[21888],
     | 99.99th=[27264]
    bw (KB/s)  : min=  307, max= 3112, per=3.60%, avg=1091.45, stdev=355.55
    lat (usec) : 500=0.05%, 750=1.06%, 1000=0.88%
    lat (msec) : 2=11.63%, 4=58.75%, 10=24.79%, 20=2.75%, 50=0.08%
    lat (msec) : 100=0.01%
  cpu          : usr=0.08%, sys=0.11%, ctx=649682, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=640000/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=21805: Tue Sep 23 20:43:00 2014
  read : io=1949.9MB, bw=998.29KB/s, iops=249 , runt=2000084msec
    clat (usec): min=541 , max=39238 , avg=4005.70, stdev=2296.33
     lat (usec): min=541 , max=39239 , avg=4005.77, stdev=2296.33
    clat percentiles (usec):
     |  1.00th=[ 1272],  5.00th=[ 1816], 10.00th=[ 2192], 20.00th=[ 2576],
     | 30.00th=[ 2896], 40.00th=[ 3152], 50.00th=[ 3440], 60.00th=[ 3728],
     | 70.00th=[ 4128], 80.00th=[ 4768], 90.00th=[ 6368], 95.00th=[ 8896],
     | 99.00th=[13120], 99.50th=[14656], 99.90th=[20352], 99.95th=[22656],
     | 99.99th=[27264]
    bw (KB/s)  : min=  310, max= 1532, per=3.30%, avg=999.24, stdev=290.40
    lat (usec) : 750=0.01%, 1000=0.13%
    lat (msec) : 2=6.88%, 4=60.21%, 10=29.18%, 20=3.48%, 50=0.11%
  cpu          : usr=0.08%, sys=0.10%, ctx=503942, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=499160/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=21806: Tue Sep 23 20:43:00 2014
  read : io=2500.0MB, bw=1089.8KB/s, iops=272 , runt=2349094msec
    clat (usec): min=223 , max=43931 , avg=3669.18, stdev=2213.36
     lat (usec): min=223 , max=43931 , avg=3669.26, stdev=2213.36
    clat percentiles (usec):
     |  1.00th=[  740],  5.00th=[ 1448], 10.00th=[ 1816], 20.00th=[ 2256],
     | 30.00th=[ 2608], 40.00th=[ 2928], 50.00th=[ 3184], 60.00th=[ 3504],
     | 70.00th=[ 3888], 80.00th=[ 4448], 90.00th=[ 5792], 95.00th=[ 8096],
     | 99.00th=[12608], 99.50th=[14144], 99.90th=[19328], 99.95th=[21632],
     | 99.99th=[27008]
    bw (KB/s)  : min=  317, max= 3648, per=3.60%, avg=1090.75, stdev=359.36
    lat (usec) : 250=0.01%, 500=0.10%, 750=1.05%, 1000=0.85%
    lat (msec) : 2=11.51%, 4=58.98%, 10=24.68%, 20=2.75%, 50=0.09%
  cpu          : usr=0.08%, sys=0.12%, ctx=648822, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=640000/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=21807: Tue Sep 23 20:43:00 2014
  read : io=2500.0MB, bw=1091.4KB/s, iops=272 , runt=2345773msec
    clat (usec): min=330 , max=40111 , avg=3663.98, stdev=2201.28
     lat (usec): min=330 , max=40111 , avg=3664.05, stdev=2201.28
    clat percentiles (usec):
     |  1.00th=[  756],  5.00th=[ 1464], 10.00th=[ 1816], 20.00th=[ 2256],
     | 30.00th=[ 2608], 40.00th=[ 2928], 50.00th=[ 3184], 60.00th=[ 3504],
     | 70.00th=[ 3888], 80.00th=[ 4448], 90.00th=[ 5792], 95.00th=[ 8032],
     | 99.00th=[12608], 99.50th=[14016], 99.90th=[19072], 99.95th=[21632],
     | 99.99th=[27008]
    bw (KB/s)  : min=  276, max= 2306, per=3.61%, avg=1092.40, stdev=353.25
    lat (usec) : 500=0.02%, 750=0.95%, 1000=0.90%
    lat (msec) : 2=11.59%, 4=59.08%, 10=24.64%, 20=2.73%, 50=0.08%
  cpu          : usr=0.08%, sys=0.11%, ctx=647368, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=640000/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=21808: Tue Sep 23 20:43:00 2014
  read : io=2500.0MB, bw=1091.9KB/s, iops=272 , runt=2344732msec
    clat (usec): min=337 , max=61181 , avg=3662.32, stdev=2202.46
     lat (usec): min=337 , max=61181 , avg=3662.39, stdev=2202.46
    clat percentiles (usec):
     |  1.00th=[  756],  5.00th=[ 1448], 10.00th=[ 1816], 20.00th=[ 2256],
     | 30.00th=[ 2608], 40.00th=[ 2928], 50.00th=[ 3184], 60.00th=[ 3504],
     | 70.00th=[ 3888], 80.00th=[ 4448], 90.00th=[ 5792], 95.00th=[ 8032],
     | 99.00th=[12480], 99.50th=[14016], 99.90th=[19328], 99.95th=[21888],
     | 99.99th=[27520]
    bw (KB/s)  : min=  307, max= 2304, per=3.61%, avg=1093.01, stdev=353.86
    lat (usec) : 500=0.02%, 750=0.96%, 1000=0.89%
    lat (msec) : 2=11.66%, 4=59.00%, 10=24.66%, 20=2.72%, 50=0.09%
    lat (msec) : 100=0.01%
  cpu          : usr=0.08%, sys=0.12%, ctx=650009, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=640000/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=21809: Tue Sep 23 20:43:00 2014
  read : io=2500.0MB, bw=1091.4KB/s, iops=272 , runt=2345837msec
    clat (usec): min=333 , max=42121 , avg=3664.11, stdev=2201.95
     lat (usec): min=333 , max=42121 , avg=3664.19, stdev=2201.95
    clat percentiles (usec):
     |  1.00th=[  748],  5.00th=[ 1448], 10.00th=[ 1816], 20.00th=[ 2256],
     | 30.00th=[ 2608], 40.00th=[ 2928], 50.00th=[ 3184], 60.00th=[ 3504],
     | 70.00th=[ 3888], 80.00th=[ 4448], 90.00th=[ 5792], 95.00th=[ 8032],
     | 99.00th=[12480], 99.50th=[14016], 99.90th=[19072], 99.95th=[21632],
     | 99.99th=[26752]
    bw (KB/s)  : min=  300, max= 2328, per=3.61%, avg=1092.33, stdev=354.34
    lat (usec) : 500=0.02%, 750=0.99%, 1000=0.91%
    lat (msec) : 2=11.67%, 4=58.89%, 10=24.68%, 20=2.75%, 50=0.08%
  cpu          : usr=0.08%, sys=0.12%, ctx=653146, majf=0, minf=26
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=640000/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=21810: Tue Sep 23 20:43:00 2014
  read : io=1996.2MB, bw=1002.4KB/s, iops=250 , runt=2039267msec
    clat (usec): min=372 , max=63554 , avg=3989.47, stdev=2310.30
     lat (usec): min=372 , max=63554 , avg=3989.54, stdev=2310.31
    clat percentiles (usec):
     |  1.00th=[ 1112],  5.00th=[ 1704], 10.00th=[ 2160], 20.00th=[ 2576],
     | 30.00th=[ 2896], 40.00th=[ 3152], 50.00th=[ 3408], 60.00th=[ 3728],
     | 70.00th=[ 4128], 80.00th=[ 4768], 90.00th=[ 6368], 95.00th=[ 8896],
     | 99.00th=[13120], 99.50th=[14656], 99.90th=[20096], 99.95th=[22912],
     | 99.99th=[28032]
    bw (KB/s)  : min=  310, max= 2408, per=3.31%, avg=1003.31, stdev=296.96
    lat (usec) : 500=0.01%, 750=0.18%, 1000=0.34%
    lat (msec) : 2=7.32%, 4=59.48%, 10=29.11%, 20=3.47%, 50=0.10%
    lat (msec) : 100=0.01%
  cpu          : usr=0.07%, sys=0.11%, ctx=522819, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=511004/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=21811: Tue Sep 23 20:43:00 2014
  read : io=1880.6MB, bw=987.57KB/s, iops=246 , runt=1949940msec
    clat (usec): min=703 , max=56462 , avg=4049.23, stdev=2306.59
     lat (usec): min=703 , max=56462 , avg=4049.30, stdev=2306.59
    clat percentiles (usec):
     |  1.00th=[ 1432],  5.00th=[ 1880], 10.00th=[ 2224], 20.00th=[ 2640],
     | 30.00th=[ 2928], 40.00th=[ 3184], 50.00th=[ 3440], 60.00th=[ 3728],
     | 70.00th=[ 4128], 80.00th=[ 4768], 90.00th=[ 6432], 95.00th=[ 9024],
     | 99.00th=[13120], 99.50th=[14784], 99.90th=[20352], 99.95th=[22656],
     | 99.99th=[28032]
    bw (KB/s)  : min=  298, max= 1540, per=3.26%, avg=988.53, stdev=288.30
    lat (usec) : 750=0.01%, 1000=0.05%
    lat (msec) : 2=5.93%, 4=60.77%, 10=29.56%, 20=3.58%, 50=0.10%
    lat (msec) : 100=0.01%
  cpu          : usr=0.07%, sys=0.10%, ctx=485042, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=481416/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=21812: Tue Sep 23 20:43:00 2014
  read : io=2500.0MB, bw=1090.7KB/s, iops=272 , runt=2347311msec
    clat (usec): min=236 , max=56896 , avg=3666.46, stdev=2208.43
     lat (usec): min=236 , max=56896 , avg=3666.53, stdev=2208.43
    clat percentiles (usec):
     |  1.00th=[  740],  5.00th=[ 1448], 10.00th=[ 1816], 20.00th=[ 2256],
     | 30.00th=[ 2608], 40.00th=[ 2928], 50.00th=[ 3184], 60.00th=[ 3504],
     | 70.00th=[ 3888], 80.00th=[ 4448], 90.00th=[ 5792], 95.00th=[ 8032],
     | 99.00th=[12608], 99.50th=[14016], 99.90th=[19072], 99.95th=[21888],
     | 99.99th=[26496]
    bw (KB/s)  : min=  320, max= 3224, per=3.60%, avg=1091.66, stdev=356.51
    lat (usec) : 250=0.01%, 500=0.06%, 750=1.03%, 1000=0.89%
    lat (msec) : 2=11.64%, 4=58.83%, 10=24.70%, 20=2.76%, 50=0.08%
    lat (msec) : 100=0.01%
  cpu          : usr=0.08%, sys=0.12%, ctx=644925, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=640000/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=21813: Tue Sep 23 20:43:00 2014
  read : io=2500.0MB, bw=1089.8KB/s, iops=272 , runt=2349176msec
    clat (usec): min=234 , max=48261 , avg=3669.34, stdev=2223.70
     lat (usec): min=234 , max=48261 , avg=3669.41, stdev=2223.70
    clat percentiles (usec):
     |  1.00th=[  732],  5.00th=[ 1448], 10.00th=[ 1816], 20.00th=[ 2256],
     | 30.00th=[ 2608], 40.00th=[ 2928], 50.00th=[ 3184], 60.00th=[ 3504],
     | 70.00th=[ 3888], 80.00th=[ 4448], 90.00th=[ 5792], 95.00th=[ 8096],
     | 99.00th=[12608], 99.50th=[14144], 99.90th=[19072], 99.95th=[21632],
     | 99.99th=[27008]
    bw (KB/s)  : min=  309, max= 3824, per=3.60%, avg=1091.04, stdev=364.64
    lat (usec) : 250=0.01%, 500=0.12%, 750=1.06%, 1000=0.87%
    lat (msec) : 2=11.60%, 4=58.77%, 10=24.68%, 20=2.82%, 50=0.08%
  cpu          : usr=0.09%, sys=0.12%, ctx=662762, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=640000/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=21814: Tue Sep 23 20:43:00 2014
  read : io=2500.0MB, bw=1090.2KB/s, iops=272 , runt=2346487msec
    clat (usec): min=273 , max=46613 , avg=3665.15, stdev=2198.81
     lat (usec): min=273 , max=46613 , avg=3665.22, stdev=2198.81
    clat percentiles (usec):
     |  1.00th=[  748],  5.00th=[ 1448], 10.00th=[ 1816], 20.00th=[ 2256],
     | 30.00th=[ 2608], 40.00th=[ 2928], 50.00th=[ 3184], 60.00th=[ 3504],
     | 70.00th=[ 3888], 80.00th=[ 4448], 90.00th=[ 5792], 95.00th=[ 7968],
     | 99.00th=[12480], 99.50th=[14016], 99.90th=[19328], 99.95th=[22144],
     | 99.99th=[26752]
    bw (KB/s)  : min=  313, max= 2408, per=3.60%, avg=1091.89, stdev=352.69
    lat (usec) : 500=0.03%, 750=0.98%, 1000=0.89%
    lat (msec) : 2=11.55%, 4=59.00%, 10=24.76%, 20=2.71%, 50=0.09%
  cpu          : usr=0.08%, sys=0.12%, ctx=647293, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=640000/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=21815: Tue Sep 23 20:43:00 2014
  read : io=2500.0MB, bw=1089.8KB/s, iops=272 , runt=2349103msec
    clat (usec): min=226 , max=71302 , avg=3669.25, stdev=2208.82
     lat (usec): min=226 , max=71302 , avg=3669.32, stdev=2208.83
    clat percentiles (usec):
     |  1.00th=[  740],  5.00th=[ 1448], 10.00th=[ 1816], 20.00th=[ 2256],
     | 30.00th=[ 2608], 40.00th=[ 2928], 50.00th=[ 3184], 60.00th=[ 3504],
     | 70.00th=[ 3888], 80.00th=[ 4448], 90.00th=[ 5792], 95.00th=[ 8032],
     | 99.00th=[12608], 99.50th=[14016], 99.90th=[19072], 99.95th=[21888],
     | 99.99th=[27008]
    bw (KB/s)  : min=  312, max= 3984, per=3.60%, avg=1090.90, stdev=360.16
    lat (usec) : 250=0.01%, 500=0.11%, 750=1.05%, 1000=0.88%
    lat (msec) : 2=11.47%, 4=58.80%, 10=24.86%, 20=2.76%, 50=0.08%
    lat (msec) : 100=0.01%
  cpu          : usr=0.08%, sys=0.12%, ctx=651377, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=640000/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=21816: Tue Sep 23 20:43:00 2014
  read : io=2500.0MB, bw=1090.1KB/s, iops=272 , runt=2346703msec
    clat (usec): min=229 , max=49687 , avg=3665.49, stdev=2203.43
     lat (usec): min=229 , max=49687 , avg=3665.56, stdev=2203.44
    clat percentiles (usec):
     |  1.00th=[  740],  5.00th=[ 1448], 10.00th=[ 1816], 20.00th=[ 2256],
     | 30.00th=[ 2608], 40.00th=[ 2928], 50.00th=[ 3184], 60.00th=[ 3504],
     | 70.00th=[ 3888], 80.00th=[ 4448], 90.00th=[ 5792], 95.00th=[ 8032],
     | 99.00th=[12608], 99.50th=[14144], 99.90th=[19328], 99.95th=[21632],
     | 99.99th=[27008]
    bw (KB/s)  : min=  310, max= 2824, per=3.61%, avg=1092.01, stdev=355.09
    lat (usec) : 250=0.01%, 500=0.04%, 750=1.03%, 1000=0.89%
    lat (msec) : 2=11.62%, 4=58.82%, 10=24.81%, 20=2.71%, 50=0.08%
  cpu          : usr=0.08%, sys=0.12%, ctx=655230, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=640000/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=21817: Tue Sep 23 20:43:00 2014
  read : io=1941.1MB, bw=995.31KB/s, iops=248 , runt=1998442msec
    clat (usec): min=408 , max=43644 , avg=4018.75, stdev=2301.73
     lat (usec): min=408 , max=43644 , avg=4018.83, stdev=2301.74
    clat percentiles (usec):
     |  1.00th=[ 1288],  5.00th=[ 1832], 10.00th=[ 2192], 20.00th=[ 2608],
     | 30.00th=[ 2928], 40.00th=[ 3152], 50.00th=[ 3440], 60.00th=[ 3728],
     | 70.00th=[ 4128], 80.00th=[ 4768], 90.00th=[ 6368], 95.00th=[ 8896],
     | 99.00th=[13120], 99.50th=[14656], 99.90th=[20096], 99.95th=[22656],
     | 99.99th=[27520]
    bw (KB/s)  : min=  310, max= 1540, per=3.29%, avg=996.08, stdev=289.99
    lat (usec) : 500=0.01%, 750=0.01%, 1000=0.12%
    lat (msec) : 2=6.58%, 4=60.40%, 10=29.27%, 20=3.51%, 50=0.10%
  cpu          : usr=0.07%, sys=0.11%, ctx=505570, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=497128/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=21818: Tue Sep 23 20:43:00 2014
  read : io=2500.0MB, bw=1088.4KB/s, iops=272 , runt=2352305msec
    clat (usec): min=222 , max=57564 , avg=3674.23, stdev=2229.30
     lat (usec): min=222 , max=57564 , avg=3674.30, stdev=2229.30
    clat percentiles (usec):
     |  1.00th=[  724],  5.00th=[ 1448], 10.00th=[ 1816], 20.00th=[ 2256],
     | 30.00th=[ 2608], 40.00th=[ 2928], 50.00th=[ 3184], 60.00th=[ 3504],
     | 70.00th=[ 3888], 80.00th=[ 4448], 90.00th=[ 5792], 95.00th=[ 8160],
     | 99.00th=[12608], 99.50th=[14144], 99.90th=[19072], 99.95th=[21888],
     | 99.99th=[27264]
    bw (KB/s)  : min=  303, max= 4272, per=3.60%, avg=1089.51, stdev=373.42
    lat (usec) : 250=0.01%, 500=0.33%, 750=1.06%, 1000=0.84%
    lat (msec) : 2=11.41%, 4=58.61%, 10=24.86%, 20=2.81%, 50=0.08%
    lat (msec) : 100=0.01%
  cpu          : usr=0.08%, sys=0.12%, ctx=652731, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=640000/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=21819: Tue Sep 23 20:43:00 2014
  read : io=1961.1MB, bw=1001.8KB/s, iops=250 , runt=2005539msec
    clat (usec): min=305 , max=59203 , avg=3991.92, stdev=2282.69
     lat (usec): min=305 , max=59203 , avg=3991.99, stdev=2282.70
    clat percentiles (usec):
     |  1.00th=[ 1272],  5.00th=[ 1816], 10.00th=[ 2192], 20.00th=[ 2576],
     | 30.00th=[ 2896], 40.00th=[ 3152], 50.00th=[ 3408], 60.00th=[ 3728],
     | 70.00th=[ 4128], 80.00th=[ 4768], 90.00th=[ 6304], 95.00th=[ 8768],
     | 99.00th=[12992], 99.50th=[14656], 99.90th=[20096], 99.95th=[22656],
     | 99.99th=[28032]
    bw (KB/s)  : min=  313, max= 1655, per=3.31%, avg=1002.77, stdev=289.37
    lat (usec) : 500=0.01%, 750=0.02%, 1000=0.15%
    lat (msec) : 2=6.90%, 4=60.37%, 10=29.07%, 20=3.39%, 50=0.10%
    lat (msec) : 100=0.01%
  cpu          : usr=0.08%, sys=0.11%, ctx=509286, majf=0, minf=26
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=502244/w=0/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
   READ: io=70541MB, aggrb=30290KB/s, minb=962KB/s, maxb=1099KB/s, mint=1948309msec, maxt=2384725msec

sudo mkdir -p /backup/fio && sudo fio randwrite4k30.fio && sudo rm -rf /backup/fio
randwrite4k30: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=1
...
randwrite4k30: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=1
fio-2.0.13
Starting 30 processes
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
Jobs: 9 (f=9): [____w_w___w_____wwww______w_w_] [100.0% done] [0K/12216K/0K /s] [0 /3054 /0  iops] [eta 00m:02s]
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=1773: Wed Sep 24 01:58:59 2014
  write: io=1941.1MB, bw=272482 B/s, iops=66 , runt=7472856msec
    clat (msec): min=1 , max=395 , avg=15.03, stdev= 7.03
     lat (msec): min=1 , max=395 , avg=15.03, stdev= 7.03
    clat percentiles (msec):
     |  1.00th=[    8],  5.00th=[    9], 10.00th=[   10], 20.00th=[   10],
     | 30.00th=[   11], 40.00th=[   12], 50.00th=[   13], 60.00th=[   14],
     | 70.00th=[   17], 80.00th=[   21], 90.00th=[   25], 95.00th=[   28],
     | 99.00th=[   38], 99.50th=[   43], 99.90th=[   52], 99.95th=[   57],
     | 99.99th=[  157]
    bw (KB/s)  : min=   75, max=  471, per=3.26%, avg=266.50, stdev=93.51
    lat (msec) : 2=0.01%, 4=0.01%, 10=20.53%, 20=57.92%, 50=21.41%
    lat (msec) : 100=0.12%, 250=0.02%, 500=0.01%
  cpu          : usr=0.03%, sys=0.03%, ctx=512504, majf=0, minf=22
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=497124/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=1774: Wed Sep 24 01:58:59 2014
  write: io=2500.0MB, bw=296666 B/s, iops=72 , runt=8836307msec
    clat (msec): min=1 , max=397 , avg=13.81, stdev= 6.94
     lat (msec): min=1 , max=397 , avg=13.81, stdev= 6.94
    clat percentiles (msec):
     |  1.00th=[    7],  5.00th=[    8], 10.00th=[    8], 20.00th=[    9],
     | 30.00th=[   10], 40.00th=[   11], 50.00th=[   12], 60.00th=[   13],
     | 70.00th=[   15], 80.00th=[   19], 90.00th=[   24], 95.00th=[   28],
     | 99.00th=[   37], 99.50th=[   42], 99.90th=[   51], 99.95th=[   56],
     | 99.99th=[  141]
    bw (KB/s)  : min=   46, max=  634, per=3.55%, avg=290.27, stdev=111.37
    lat (msec) : 2=0.01%, 4=0.01%, 10=32.54%, 20=49.92%, 50=17.42%
    lat (msec) : 100=0.10%, 250=0.01%, 500=0.01%
  cpu          : usr=0.03%, sys=0.03%, ctx=656191, majf=0, minf=22
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=640000/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=1775: Wed Sep 24 01:58:59 2014
  write: io=2500.0MB, bw=296664 B/s, iops=72 , runt=8836366msec
    clat (msec): min=1 , max=397 , avg=13.81, stdev= 6.94
     lat (msec): min=1 , max=397 , avg=13.81, stdev= 6.94
    clat percentiles (msec):
     |  1.00th=[    7],  5.00th=[    8], 10.00th=[    8], 20.00th=[    9],
     | 30.00th=[   10], 40.00th=[   11], 50.00th=[   12], 60.00th=[   13],
     | 70.00th=[   15], 80.00th=[   19], 90.00th=[   24], 95.00th=[   28],
     | 99.00th=[   37], 99.50th=[   42], 99.90th=[   51], 99.95th=[   56],
     | 99.99th=[  133]
    bw (KB/s)  : min=   81, max=  612, per=3.55%, avg=290.26, stdev=111.42
    lat (msec) : 2=0.01%, 4=0.01%, 10=32.50%, 20=49.97%, 50=17.41%
    lat (msec) : 100=0.10%, 250=0.01%, 500=0.01%
  cpu          : usr=0.03%, sys=0.03%, ctx=648459, majf=0, minf=22
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=640000/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=1776: Wed Sep 24 01:58:59 2014
  write: io=2500.0MB, bw=296662 B/s, iops=72 , runt=8836431msec
    clat (msec): min=1 , max=397 , avg=13.81, stdev= 6.93
     lat (msec): min=1 , max=397 , avg=13.81, stdev= 6.93
    clat percentiles (msec):
     |  1.00th=[    7],  5.00th=[    8], 10.00th=[    8], 20.00th=[    9],
     | 30.00th=[   10], 40.00th=[   11], 50.00th=[   12], 60.00th=[   13],
     | 70.00th=[   15], 80.00th=[   19], 90.00th=[   24], 95.00th=[   28],
     | 99.00th=[   37], 99.50th=[   42], 99.90th=[   51], 99.95th=[   56],
     | 99.99th=[  129]
    bw (KB/s)  : min=   68, max=  636, per=3.55%, avg=290.27, stdev=111.37
    lat (msec) : 2=0.01%, 4=0.01%, 10=32.48%, 20=50.00%, 50=17.39%
    lat (msec) : 100=0.10%, 250=0.01%, 500=0.01%
  cpu          : usr=0.03%, sys=0.03%, ctx=669421, majf=0, minf=22
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=640000/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=1777: Wed Sep 24 01:58:59 2014
  write: io=2500.0MB, bw=296643 B/s, iops=72 , runt=8836995msec
    clat (msec): min=1 , max=396 , avg=13.81, stdev= 6.94
     lat (msec): min=1 , max=396 , avg=13.81, stdev= 6.94
    clat percentiles (msec):
     |  1.00th=[    7],  5.00th=[    8], 10.00th=[    8], 20.00th=[    9],
     | 30.00th=[   10], 40.00th=[   11], 50.00th=[   12], 60.00th=[   13],
     | 70.00th=[   15], 80.00th=[   19], 90.00th=[   24], 95.00th=[   28],
     | 99.00th=[   37], 99.50th=[   42], 99.90th=[   51], 99.95th=[   55],
     | 99.99th=[  135]
    bw (KB/s)  : min=   61, max=  756, per=3.55%, avg=290.23, stdev=111.33
    lat (msec) : 2=0.01%, 4=0.02%, 10=32.41%, 20=50.04%, 50=17.41%
    lat (msec) : 100=0.09%, 250=0.01%, 500=0.01%
  cpu          : usr=0.03%, sys=0.03%, ctx=691896, majf=0, minf=22
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=640000/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=1778: Wed Sep 24 01:58:59 2014
  write: io=2500.0MB, bw=296651 B/s, iops=72 , runt=8836779msec
    clat (msec): min=1 , max=395 , avg=13.81, stdev= 6.94
     lat (msec): min=1 , max=395 , avg=13.81, stdev= 6.94
    clat percentiles (msec):
     |  1.00th=[    7],  5.00th=[    8], 10.00th=[    8], 20.00th=[    9],
     | 30.00th=[   10], 40.00th=[   11], 50.00th=[   12], 60.00th=[   13],
     | 70.00th=[   15], 80.00th=[   19], 90.00th=[   24], 95.00th=[   28],
     | 99.00th=[   37], 99.50th=[   42], 99.90th=[   51], 99.95th=[   56],
     | 99.99th=[  131]
    bw (KB/s)  : min=   47, max=  779, per=3.55%, avg=290.23, stdev=111.43
    lat (msec) : 2=0.01%, 4=0.02%, 10=32.44%, 20=50.05%, 50=17.37%
    lat (msec) : 100=0.10%, 250=0.01%, 500=0.01%
  cpu          : usr=0.03%, sys=0.03%, ctx=685294, majf=0, minf=24
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=640000/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=1779: Wed Sep 24 01:58:59 2014
  write: io=2500.0MB, bw=296643 B/s, iops=72 , runt=8836996msec
    clat (msec): min=1 , max=287 , avg=13.81, stdev= 6.93
     lat (msec): min=1 , max=287 , avg=13.81, stdev= 6.93
    clat percentiles (msec):
     |  1.00th=[    6],  5.00th=[    8], 10.00th=[    8], 20.00th=[    9],
     | 30.00th=[   10], 40.00th=[   11], 50.00th=[   12], 60.00th=[   13],
     | 70.00th=[   15], 80.00th=[   19], 90.00th=[   24], 95.00th=[   28],
     | 99.00th=[   37], 99.50th=[   42], 99.90th=[   51], 99.95th=[   56],
     | 99.99th=[  135]
    bw (KB/s)  : min=   79, max=  848, per=3.55%, avg=290.18, stdev=111.54
    lat (msec) : 2=0.01%, 4=0.02%, 10=32.50%, 20=49.92%, 50=17.43%
    lat (msec) : 100=0.10%, 250=0.01%, 500=0.01%
  cpu          : usr=0.03%, sys=0.03%, ctx=675725, majf=0, minf=22
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=640000/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=1780: Wed Sep 24 01:58:59 2014
  write: io=2500.0MB, bw=296664 B/s, iops=72 , runt=8836365msec
    clat (msec): min=1 , max=287 , avg=13.81, stdev= 6.92
     lat (msec): min=1 , max=287 , avg=13.81, stdev= 6.92
    clat percentiles (msec):
     |  1.00th=[    7],  5.00th=[    8], 10.00th=[    8], 20.00th=[    9],
     | 30.00th=[   10], 40.00th=[   11], 50.00th=[   12], 60.00th=[   13],
     | 70.00th=[   15], 80.00th=[   19], 90.00th=[   24], 95.00th=[   28],
     | 99.00th=[   37], 99.50th=[   41], 99.90th=[   51], 99.95th=[   56],
     | 99.99th=[  133]
    bw (KB/s)  : min=   45, max=  619, per=3.55%, avg=290.28, stdev=111.51
    lat (msec) : 2=0.01%, 4=0.01%, 10=32.51%, 20=49.97%, 50=17.40%
    lat (msec) : 100=0.10%, 250=0.01%, 500=0.01%
  cpu          : usr=0.03%, sys=0.03%, ctx=664165, majf=0, minf=22
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=640000/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=1781: Wed Sep 24 01:58:59 2014
  write: io=1890.8MB, bw=271487 B/s, iops=66 , runt=7302826msec
    clat (msec): min=1 , max=397 , avg=15.09, stdev= 7.04
     lat (msec): min=1 , max=397 , avg=15.09, stdev= 7.04
    clat percentiles (msec):
     |  1.00th=[    8],  5.00th=[    9], 10.00th=[   10], 20.00th=[   11],
     | 30.00th=[   11], 40.00th=[   12], 50.00th=[   13], 60.00th=[   14],
     | 70.00th=[   17], 80.00th=[   21], 90.00th=[   25], 95.00th=[   28],
     | 99.00th=[   38], 99.50th=[   43], 99.90th=[   52], 99.95th=[   57],
     | 99.99th=[  161]
    bw (KB/s)  : min=   51, max=  468, per=3.25%, avg=265.57, stdev=93.08
    lat (msec) : 2=0.01%, 4=0.01%, 10=20.07%, 20=58.22%, 50=21.57%
    lat (msec) : 100=0.12%, 250=0.02%, 500=0.01%
  cpu          : usr=0.03%, sys=0.03%, ctx=489578, majf=0, minf=22
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=484040/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=1782: Wed Sep 24 01:58:59 2014
  write: io=2500.0MB, bw=296681 B/s, iops=72 , runt=8835887msec
    clat (msec): min=1 , max=397 , avg=13.80, stdev= 6.93
     lat (msec): min=1 , max=397 , avg=13.80, stdev= 6.93
    clat percentiles (msec):
     |  1.00th=[    7],  5.00th=[    8], 10.00th=[    8], 20.00th=[    9],
     | 30.00th=[   10], 40.00th=[   11], 50.00th=[   12], 60.00th=[   13],
     | 70.00th=[   15], 80.00th=[   19], 90.00th=[   24], 95.00th=[   28],
     | 99.00th=[   37], 99.50th=[   42], 99.90th=[   51], 99.95th=[   56],
     | 99.99th=[  131]
    bw (KB/s)  : min=   53, max=  623, per=3.55%, avg=290.28, stdev=111.51
    lat (msec) : 2=0.01%, 4=0.01%, 10=32.53%, 20=49.97%, 50=17.39%
    lat (msec) : 100=0.09%, 250=0.01%, 500=0.01%
  cpu          : usr=0.03%, sys=0.03%, ctx=644882, majf=0, minf=24
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=640000/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=1783: Wed Sep 24 01:58:59 2014
  write: io=2500.0MB, bw=296644 B/s, iops=72 , runt=8836975msec
    clat (msec): min=1 , max=396 , avg=13.81, stdev= 6.95
     lat (msec): min=1 , max=396 , avg=13.81, stdev= 6.95
    clat percentiles (msec):
     |  1.00th=[    6],  5.00th=[    8], 10.00th=[    8], 20.00th=[    9],
     | 30.00th=[   10], 40.00th=[   11], 50.00th=[   12], 60.00th=[   13],
     | 70.00th=[   15], 80.00th=[   19], 90.00th=[   24], 95.00th=[   28],
     | 99.00th=[   37], 99.50th=[   42], 99.90th=[   51], 99.95th=[   56],
     | 99.99th=[  141]
    bw (KB/s)  : min=   54, max=  838, per=3.55%, avg=290.20, stdev=111.56
    lat (msec) : 2=0.01%, 4=0.02%, 10=32.47%, 20=49.99%, 50=17.40%
    lat (msec) : 100=0.10%, 250=0.01%, 500=0.01%
  cpu          : usr=0.03%, sys=0.03%, ctx=673171, majf=0, minf=24
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=640000/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=1784: Wed Sep 24 01:58:59 2014
  write: io=2500.0MB, bw=296655 B/s, iops=72 , runt=8836653msec
    clat (msec): min=1 , max=388 , avg=13.81, stdev= 6.93
     lat (msec): min=1 , max=388 , avg=13.81, stdev= 6.93
    clat percentiles (msec):
     |  1.00th=[    6],  5.00th=[    8], 10.00th=[    8], 20.00th=[    9],
     | 30.00th=[   10], 40.00th=[   11], 50.00th=[   12], 60.00th=[   13],
     | 70.00th=[   15], 80.00th=[   19], 90.00th=[   24], 95.00th=[   28],
     | 99.00th=[   37], 99.50th=[   42], 99.90th=[   51], 99.95th=[   55],
     | 99.99th=[  143]
    bw (KB/s)  : min=   46, max=  614, per=3.55%, avg=290.26, stdev=111.40
    lat (msec) : 2=0.01%, 4=0.01%, 10=32.42%, 20=50.05%, 50=17.41%
    lat (msec) : 100=0.09%, 250=0.01%, 500=0.01%
  cpu          : usr=0.03%, sys=0.03%, ctx=688222, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=640000/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=1785: Wed Sep 24 01:58:59 2014
  write: io=2500.0MB, bw=296657 B/s, iops=72 , runt=8836582msec
    clat (msec): min=1 , max=389 , avg=13.81, stdev= 6.94
     lat (msec): min=1 , max=389 , avg=13.81, stdev= 6.94
    clat percentiles (msec):
     |  1.00th=[    7],  5.00th=[    8], 10.00th=[    8], 20.00th=[    9],
     | 30.00th=[   10], 40.00th=[   11], 50.00th=[   12], 60.00th=[   13],
     | 70.00th=[   15], 80.00th=[   19], 90.00th=[   24], 95.00th=[   28],
     | 99.00th=[   37], 99.50th=[   42], 99.90th=[   51], 99.95th=[   56],
     | 99.99th=[  135]
    bw (KB/s)  : min=   74, max=  621, per=3.55%, avg=290.27, stdev=111.39
    lat (msec) : 2=0.01%, 4=0.01%, 10=32.48%, 20=50.00%, 50=17.41%
    lat (msec) : 100=0.09%, 250=0.01%, 500=0.01%
  cpu          : usr=0.03%, sys=0.03%, ctx=661838, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=640000/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=1786: Wed Sep 24 01:58:59 2014
  write: io=1978.4MB, bw=272114 B/s, iops=66 , runt=7623555msec
    clat (msec): min=1 , max=397 , avg=15.05, stdev= 7.10
     lat (msec): min=1 , max=397 , avg=15.05, stdev= 7.10
    clat percentiles (msec):
     |  1.00th=[    8],  5.00th=[    9], 10.00th=[   10], 20.00th=[   10],
     | 30.00th=[   11], 40.00th=[   12], 50.00th=[   13], 60.00th=[   14],
     | 70.00th=[   17], 80.00th=[   21], 90.00th=[   25], 95.00th=[   28],
     | 99.00th=[   39], 99.50th=[   43], 99.90th=[   52], 99.95th=[   57],
     | 99.99th=[  157]
    bw (KB/s)  : min=   77, max=  536, per=3.26%, avg=266.17, stdev=94.71
    lat (msec) : 2=0.01%, 4=0.01%, 10=21.11%, 20=57.18%, 50=21.57%
    lat (msec) : 100=0.12%, 250=0.02%, 500=0.01%
  cpu          : usr=0.03%, sys=0.03%, ctx=508596, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=506464/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=1787: Wed Sep 24 01:58:59 2014
  write: io=2500.0MB, bw=296694 B/s, iops=72 , runt=8835496msec
    clat (msec): min=1 , max=293 , avg=13.80, stdev= 6.93
     lat (msec): min=1 , max=293 , avg=13.80, stdev= 6.93
    clat percentiles (msec):
     |  1.00th=[    7],  5.00th=[    8], 10.00th=[    8], 20.00th=[    9],
     | 30.00th=[   10], 40.00th=[   11], 50.00th=[   12], 60.00th=[   13],
     | 70.00th=[   15], 80.00th=[   19], 90.00th=[   24], 95.00th=[   28],
     | 99.00th=[   37], 99.50th=[   42], 99.90th=[   51], 99.95th=[   56],
     | 99.99th=[  141]
    bw (KB/s)  : min=   79, max=  637, per=3.55%, avg=290.31, stdev=111.47
    lat (msec) : 2=0.01%, 4=0.01%, 10=32.49%, 20=50.01%, 50=17.38%
    lat (msec) : 100=0.10%, 250=0.01%, 500=0.01%
  cpu          : usr=0.03%, sys=0.03%, ctx=653225, majf=0, minf=24
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=640000/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=1788: Wed Sep 24 01:58:59 2014
  write: io=1949.9MB, bw=272451 B/s, iops=66 , runt=7504317msec
    clat (msec): min=1 , max=397 , avg=15.03, stdev= 7.03
     lat (msec): min=1 , max=397 , avg=15.03, stdev= 7.03
    clat percentiles (msec):
     |  1.00th=[    8],  5.00th=[    9], 10.00th=[   10], 20.00th=[   10],
     | 30.00th=[   11], 40.00th=[   12], 50.00th=[   13], 60.00th=[   14],
     | 70.00th=[   17], 80.00th=[   21], 90.00th=[   25], 95.00th=[   28],
     | 99.00th=[   38], 99.50th=[   42], 99.90th=[   52], 99.95th=[   57],
     | 99.99th=[  161]
    bw (KB/s)  : min=   73, max=  498, per=3.26%, avg=266.47, stdev=93.63
    lat (msec) : 2=0.01%, 4=0.01%, 10=20.58%, 20=57.84%, 50=21.44%
    lat (msec) : 100=0.11%, 250=0.02%, 500=0.01%
  cpu          : usr=0.03%, sys=0.03%, ctx=519902, majf=0, minf=24
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=499160/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=1789: Wed Sep 24 01:58:59 2014
  write: io=2500.0MB, bw=296645 B/s, iops=72 , runt=8836939msec
    clat (msec): min=1 , max=396 , avg=13.81, stdev= 6.94
     lat (msec): min=1 , max=396 , avg=13.81, stdev= 6.94
    clat percentiles (msec):
     |  1.00th=[    7],  5.00th=[    8], 10.00th=[    8], 20.00th=[    9],
     | 30.00th=[   10], 40.00th=[   11], 50.00th=[   12], 60.00th=[   13],
     | 70.00th=[   15], 80.00th=[   19], 90.00th=[   24], 95.00th=[   28],
     | 99.00th=[   37], 99.50th=[   42], 99.90th=[   51], 99.95th=[   55],
     | 99.99th=[  139]
    bw (KB/s)  : min=   79, max=  796, per=3.55%, avg=290.22, stdev=111.48
    lat (msec) : 2=0.01%, 4=0.02%, 10=32.50%, 20=49.97%, 50=17.40%
    lat (msec) : 100=0.09%, 250=0.01%, 500=0.01%
  cpu          : usr=0.03%, sys=0.03%, ctx=667158, majf=0, minf=24
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=640000/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=1790: Wed Sep 24 01:58:59 2014
  write: io=2500.0MB, bw=296646 B/s, iops=72 , runt=8836909msec
    clat (msec): min=1 , max=395 , avg=13.81, stdev= 6.94
     lat (msec): min=1 , max=395 , avg=13.81, stdev= 6.94
    clat percentiles (msec):
     |  1.00th=[    7],  5.00th=[    8], 10.00th=[    8], 20.00th=[    9],
     | 30.00th=[   10], 40.00th=[   11], 50.00th=[   12], 60.00th=[   13],
     | 70.00th=[   15], 80.00th=[   19], 90.00th=[   24], 95.00th=[   28],
     | 99.00th=[   37], 99.50th=[   41], 99.90th=[   51], 99.95th=[   56],
     | 99.99th=[  137]
    bw (KB/s)  : min=   46, max=  874, per=3.55%, avg=290.22, stdev=111.60
    lat (msec) : 2=0.01%, 4=0.02%, 10=32.50%, 20=49.92%, 50=17.45%
    lat (msec) : 100=0.10%, 250=0.01%, 500=0.01%
  cpu          : usr=0.03%, sys=0.03%, ctx=682874, majf=0, minf=24
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=640000/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=1791: Wed Sep 24 01:58:59 2014
  write: io=2500.0MB, bw=296647 B/s, iops=72 , runt=8836895msec
    clat (msec): min=1 , max=383 , avg=13.81, stdev= 6.94
     lat (msec): min=1 , max=383 , avg=13.81, stdev= 6.94
    clat percentiles (msec):
     |  1.00th=[    7],  5.00th=[    8], 10.00th=[    8], 20.00th=[    9],
     | 30.00th=[   10], 40.00th=[   11], 50.00th=[   12], 60.00th=[   13],
     | 70.00th=[   15], 80.00th=[   19], 90.00th=[   24], 95.00th=[   28],
     | 99.00th=[   37], 99.50th=[   42], 99.90th=[   51], 99.95th=[   56],
     | 99.99th=[  139]
    bw (KB/s)  : min=   53, max=  628, per=3.55%, avg=290.17, stdev=111.47
    lat (msec) : 2=0.01%, 4=0.02%, 10=32.52%, 20=49.93%, 50=17.41%
    lat (msec) : 100=0.10%, 250=0.01%, 500=0.01%
  cpu          : usr=0.03%, sys=0.03%, ctx=681355, majf=0, minf=24
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=640000/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=1792: Wed Sep 24 01:58:59 2014
  write: io=2500.0MB, bw=296645 B/s, iops=72 , runt=8836936msec
    clat (msec): min=1 , max=397 , avg=13.81, stdev= 6.94
     lat (msec): min=1 , max=397 , avg=13.81, stdev= 6.94
    clat percentiles (msec):
     |  1.00th=[    7],  5.00th=[    8], 10.00th=[    8], 20.00th=[    9],
     | 30.00th=[   10], 40.00th=[   11], 50.00th=[   12], 60.00th=[   13],
     | 70.00th=[   15], 80.00th=[   19], 90.00th=[   24], 95.00th=[   28],
     | 99.00th=[   37], 99.50th=[   42], 99.90th=[   51], 99.95th=[   55],
     | 99.99th=[  141]
    bw (KB/s)  : min=   72, max=  864, per=3.55%, avg=290.23, stdev=111.43
    lat (msec) : 2=0.01%, 4=0.02%, 10=32.54%, 20=49.89%, 50=17.43%
    lat (msec) : 100=0.09%, 250=0.01%, 500=0.01%
  cpu          : usr=0.03%, sys=0.03%, ctx=665378, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=640000/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=1793: Wed Sep 24 01:58:59 2014
  write: io=1996.2MB, bw=273054 B/s, iops=66 , runt=7665393msec
    clat (msec): min=1 , max=397 , avg=15.00, stdev= 7.10
     lat (msec): min=1 , max=397 , avg=15.00, stdev= 7.10
    clat percentiles (msec):
     |  1.00th=[    8],  5.00th=[    9], 10.00th=[    9], 20.00th=[   10],
     | 30.00th=[   11], 40.00th=[   12], 50.00th=[   13], 60.00th=[   14],
     | 70.00th=[   17], 80.00th=[   21], 90.00th=[   25], 95.00th=[   28],
     | 99.00th=[   39], 99.50th=[   43], 99.90th=[   52], 99.95th=[   57],
     | 99.99th=[  161]
    bw (KB/s)  : min=   47, max=  578, per=3.27%, avg=267.08, stdev=95.71
    lat (msec) : 2=0.01%, 4=0.01%, 10=21.57%, 20=56.91%, 50=21.38%
    lat (msec) : 100=0.12%, 250=0.02%, 500=0.01%
  cpu          : usr=0.03%, sys=0.03%, ctx=536143, majf=0, minf=23
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=511004/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=1794: Wed Sep 24 01:58:59 2014
  write: io=1880.6MB, bw=271279 B/s, iops=66 , runt=7268800msec
    clat (msec): min=1 , max=395 , avg=15.10, stdev= 7.05
     lat (msec): min=1 , max=395 , avg=15.10, stdev= 7.05
    clat percentiles (msec):
     |  1.00th=[    8],  5.00th=[    9], 10.00th=[   10], 20.00th=[   11],
     | 30.00th=[   11], 40.00th=[   12], 50.00th=[   13], 60.00th=[   15],
     | 70.00th=[   17], 80.00th=[   21], 90.00th=[   25], 95.00th=[   28],
     | 99.00th=[   38], 99.50th=[   43], 99.90th=[   52], 99.95th=[   57],
     | 99.99th=[  163]
    bw (KB/s)  : min=   80, max=  458, per=3.25%, avg=265.34, stdev=93.15
    lat (msec) : 2=0.01%, 4=0.01%, 10=20.01%, 20=58.14%, 50=21.71%
    lat (msec) : 100=0.12%, 250=0.02%, 500=0.01%
  cpu          : usr=0.03%, sys=0.03%, ctx=485729, majf=0, minf=22
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=481416/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=1795: Wed Sep 24 01:58:59 2014
  write: io=2500.0MB, bw=296656 B/s, iops=72 , runt=8836628msec
    clat (msec): min=1 , max=291 , avg=13.81, stdev= 6.93
     lat (msec): min=1 , max=291 , avg=13.81, stdev= 6.93
    clat percentiles (msec):
     |  1.00th=[    7],  5.00th=[    8], 10.00th=[    8], 20.00th=[    9],
     | 30.00th=[   10], 40.00th=[   11], 50.00th=[   12], 60.00th=[   13],
     | 70.00th=[   15], 80.00th=[   19], 90.00th=[   24], 95.00th=[   28],
     | 99.00th=[   37], 99.50th=[   42], 99.90th=[   51], 99.95th=[   55],
     | 99.99th=[  145]
    bw (KB/s)  : min=   48, max=  710, per=3.55%, avg=290.27, stdev=111.46
    lat (msec) : 2=0.01%, 4=0.01%, 10=32.47%, 20=50.01%, 50=17.40%
    lat (msec) : 100=0.10%, 250=0.01%, 500=0.01%
  cpu          : usr=0.03%, sys=0.03%, ctx=658629, majf=0, minf=22
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=640000/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=1796: Wed Sep 24 01:58:59 2014
  write: io=2500.0MB, bw=296653 B/s, iops=72 , runt=8836720msec
    clat (msec): min=1 , max=383 , avg=13.81, stdev= 6.94
     lat (msec): min=1 , max=383 , avg=13.81, stdev= 6.94
    clat percentiles (msec):
     |  1.00th=[    7],  5.00th=[    8], 10.00th=[    8], 20.00th=[    9],
     | 30.00th=[   10], 40.00th=[   11], 50.00th=[   12], 60.00th=[   13],
     | 70.00th=[   15], 80.00th=[   19], 90.00th=[   24], 95.00th=[   28],
     | 99.00th=[   37], 99.50th=[   42], 99.90th=[   51], 99.95th=[   56],
     | 99.99th=[  137]
    bw (KB/s)  : min=   47, max=  704, per=3.55%, avg=290.27, stdev=111.45
    lat (msec) : 2=0.01%, 4=0.02%, 10=32.52%, 20=49.90%, 50=17.46%
    lat (msec) : 100=0.10%, 250=0.01%, 500=0.01%
  cpu          : usr=0.03%, sys=0.03%, ctx=649845, majf=0, minf=22
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=640000/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=1797: Wed Sep 24 01:58:59 2014
  write: io=2500.0MB, bw=296660 B/s, iops=72 , runt=8836494msec
    clat (msec): min=1 , max=396 , avg=13.81, stdev= 6.94
     lat (msec): min=1 , max=396 , avg=13.81, stdev= 6.94
    clat percentiles (msec):
     |  1.00th=[    7],  5.00th=[    8], 10.00th=[    8], 20.00th=[    9],
     | 30.00th=[   10], 40.00th=[   11], 50.00th=[   12], 60.00th=[   13],
     | 70.00th=[   15], 80.00th=[   19], 90.00th=[   24], 95.00th=[   28],
     | 99.00th=[   37], 99.50th=[   42], 99.90th=[   51], 99.95th=[   56],
     | 99.99th=[  141]
    bw (KB/s)  : min=   68, max=  608, per=3.55%, avg=290.25, stdev=111.40
    lat (msec) : 2=0.01%, 4=0.01%, 10=32.50%, 20=49.99%, 50=17.39%
    lat (msec) : 100=0.10%, 250=0.01%, 500=0.01%
  cpu          : usr=0.03%, sys=0.03%, ctx=652355, majf=0, minf=22
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=640000/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=1798: Wed Sep 24 01:58:59 2014
  write: io=2500.0MB, bw=296660 B/s, iops=72 , runt=8836506msec
    clat (msec): min=1 , max=389 , avg=13.81, stdev= 6.94
     lat (msec): min=1 , max=389 , avg=13.81, stdev= 6.94
    clat percentiles (msec):
     |  1.00th=[    7],  5.00th=[    8], 10.00th=[    8], 20.00th=[    9],
     | 30.00th=[   10], 40.00th=[   11], 50.00th=[   12], 60.00th=[   13],
     | 70.00th=[   15], 80.00th=[   19], 90.00th=[   24], 95.00th=[   28],
     | 99.00th=[   37], 99.50th=[   42], 99.90th=[   51], 99.95th=[   56],
     | 99.99th=[  137]
    bw (KB/s)  : min=   53, max=  611, per=3.55%, avg=290.24, stdev=111.49
    lat (msec) : 2=0.01%, 4=0.01%, 10=32.48%, 20=49.97%, 50=17.42%
    lat (msec) : 100=0.10%, 250=0.01%, 500=0.01%
  cpu          : usr=0.03%, sys=0.03%, ctx=662780, majf=0, minf=22
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=640000/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=1799: Wed Sep 24 01:58:59 2014
  write: io=2500.0MB, bw=296646 B/s, iops=72 , runt=8836904msec
    clat (msec): min=1 , max=389 , avg=13.81, stdev= 6.95
     lat (msec): min=1 , max=389 , avg=13.81, stdev= 6.95
    clat percentiles (msec):
     |  1.00th=[    6],  5.00th=[    8], 10.00th=[    8], 20.00th=[    9],
     | 30.00th=[   10], 40.00th=[   11], 50.00th=[   12], 60.00th=[   13],
     | 70.00th=[   15], 80.00th=[   19], 90.00th=[   24], 95.00th=[   28],
     | 99.00th=[   37], 99.50th=[   42], 99.90th=[   51], 99.95th=[   56],
     | 99.99th=[  139]
    bw (KB/s)  : min=   46, max=  622, per=3.55%, avg=290.20, stdev=111.53
    lat (msec) : 2=0.01%, 4=0.02%, 10=32.52%, 20=49.88%, 50=17.46%
    lat (msec) : 100=0.10%, 250=0.01%, 500=0.01%
  cpu          : usr=0.03%, sys=0.03%, ctx=657082, majf=0, minf=22
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=640000/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=1800: Wed Sep 24 01:58:59 2014
  write: io=1941.1MB, bw=272316 B/s, iops=66 , runt=7477475msec
    clat (msec): min=1 , max=381 , avg=15.04, stdev= 7.03
     lat (msec): min=1 , max=381 , avg=15.04, stdev= 7.03
    clat percentiles (msec):
     |  1.00th=[    8],  5.00th=[    9], 10.00th=[   10], 20.00th=[   10],
     | 30.00th=[   11], 40.00th=[   12], 50.00th=[   13], 60.00th=[   14],
     | 70.00th=[   17], 80.00th=[   21], 90.00th=[   25], 95.00th=[   28],
     | 99.00th=[   38], 99.50th=[   43], 99.90th=[   52], 99.95th=[   57],
     | 99.99th=[  161]
    bw (KB/s)  : min=   81, max=  479, per=3.26%, avg=266.37, stdev=93.52
    lat (msec) : 2=0.01%, 4=0.01%, 10=20.54%, 20=57.87%, 50=21.45%
    lat (msec) : 100=0.11%, 250=0.02%, 500=0.01%
  cpu          : usr=0.03%, sys=0.03%, ctx=513795, majf=0, minf=22
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=497128/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=1801: Wed Sep 24 01:58:59 2014
  write: io=2500.0MB, bw=296640 B/s, iops=72 , runt=8837098msec
    clat (usec): min=937 , max=395507 , avg=13806.25, stdev=6946.63
     lat (usec): min=937 , max=395507 , avg=13806.39, stdev=6946.68
    clat percentiles (msec):
     |  1.00th=[    7],  5.00th=[    8], 10.00th=[    8], 20.00th=[    9],
     | 30.00th=[   10], 40.00th=[   11], 50.00th=[   12], 60.00th=[   13],
     | 70.00th=[   15], 80.00th=[   19], 90.00th=[   24], 95.00th=[   28],
     | 99.00th=[   37], 99.50th=[   42], 99.90th=[   51], 99.95th=[   56],
     | 99.99th=[  145]
    bw (KB/s)  : min=   46, max=  892, per=3.55%, avg=290.17, stdev=111.45
    lat (usec) : 1000=0.01%
    lat (msec) : 2=0.02%, 4=0.02%, 10=32.42%, 20=49.99%, 50=17.43%
    lat (msec) : 100=0.10%, 250=0.01%, 500=0.01%
  cpu          : usr=0.03%, sys=0.03%, ctx=683745, majf=0, minf=22
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=640000/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=1802: Wed Sep 24 01:58:59 2014
  write: io=1961.1MB, bw=272068 B/s, iops=66 , runt=7561293msec
    clat (msec): min=1 , max=381 , avg=15.05, stdev= 7.06
     lat (msec): min=1 , max=381 , avg=15.05, stdev= 7.06
    clat percentiles (msec):
     |  1.00th=[    8],  5.00th=[    9], 10.00th=[   10], 20.00th=[   10],
     | 30.00th=[   11], 40.00th=[   12], 50.00th=[   13], 60.00th=[   14],
     | 70.00th=[   17], 80.00th=[   21], 90.00th=[   25], 95.00th=[   28],
     | 99.00th=[   38], 99.50th=[   43], 99.90th=[   52], 99.95th=[   57],
     | 99.99th=[  153]
    bw (KB/s)  : min=   47, max=  520, per=3.26%, avg=266.12, stdev=94.17
    lat (msec) : 2=0.01%, 4=0.01%, 10=20.78%, 20=57.60%, 50=21.47%
    lat (msec) : 100=0.12%, 250=0.02%, 500=0.01%
  cpu          : usr=0.03%, sys=0.03%, ctx=513048, majf=0, minf=23
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=502244/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
  WRITE: io=70541MB, aggrb=8173KB/s, minb=264KB/s, maxb=289KB/s, mint=7268800msec, maxt=8837098msec

sudo mkdir -p /backup/fio && sudo fio seqread.fio       && sudo rm -rf /backup/fio
seqread: (g=0): rw=read, bs=1M-1M/1M-1M/1M-1M, ioengine=sync, iodepth=1
fio-2.0.13
Starting 1 process
seqread: Laying out IO file(s) (1 file(s) / 10240MB)
Jobs: 1 (f=1): [R] [100.0% done] [88064K/0K/0K /s] [86 /0 /0  iops] [eta 00m:00s]
seqread: (groupid=0, jobs=1): err= 0: pid=1801: Tue Sep 23 17:58:24 2014
  read : io=51200MB, bw=87464KB/s, iops=85 , runt=599434msec
    clat (msec): min=9 , max=51 , avg=11.71, stdev= 1.70
     lat (msec): min=9 , max=51 , avg=11.71, stdev= 1.70
    clat percentiles (usec):
     |  1.00th=[10048],  5.00th=[10176], 10.00th=[10304], 20.00th=[10688],
     | 30.00th=[11072], 40.00th=[11200], 50.00th=[11328], 60.00th=[11456],
     | 70.00th=[11712], 80.00th=[12224], 90.00th=[13248], 95.00th=[14784],
     | 99.00th=[18560], 99.50th=[20352], 99.90th=[26752], 99.95th=[29824],
     | 99.99th=[34048]
    bw (KB/s)  : min=68942, max=95872, per=100.00%, avg=87541.38, stdev=3752.52
    lat (msec) : 10=1.07%, 20=98.37%, 50=0.56%, 100=0.01%
  cpu          : usr=0.03%, sys=0.50%, ctx=350872, majf=0, minf=282
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=51200/w=0/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
   READ: io=51200MB, aggrb=87463KB/s, minb=87463KB/s, maxb=87463KB/s, mint=599434msec, maxt=599434msec

sudo mkdir -p /backup/fio && sudo fio seqwrite.fio      && sudo rm -rf /backup/fio
seqwrite: (g=0): rw=write, bs=1M-1M/1M-1M/1M-1M, ioengine=sync, iodepth=1
fio-2.0.13
Starting 1 process
seqwrite: Laying out IO file(s) (1 file(s) / 10240MB)
Jobs: 1 (f=1): [W] [100.0% done] [0K/41984K/0K /s] [0 /41 /0  iops] [eta 00m:00s]
seqwrite: (groupid=0, jobs=1): err= 0: pid=1858: Tue Sep 23 18:23:06 2014
  write: io=51200MB, bw=38880KB/s, iops=37 , runt=1348477msec
    clat (msec): min=19 , max=644 , avg=26.29, stdev= 5.87
     lat (msec): min=19 , max=644 , avg=26.34, stdev= 5.87
    clat percentiles (msec):
     |  1.00th=[   23],  5.00th=[   24], 10.00th=[   24], 20.00th=[   24],
     | 30.00th=[   24], 40.00th=[   25], 50.00th=[   25], 60.00th=[   26],
     | 70.00th=[   27], 80.00th=[   29], 90.00th=[   31], 95.00th=[   34],
     | 99.00th=[   41], 99.50th=[   44], 99.90th=[   56], 99.95th=[   69],
     | 99.99th=[  235]
    bw (KB/s)  : min=16205, max=43743, per=100.00%, avg=38935.33, stdev=3476.64
    lat (msec) : 20=0.01%, 50=99.82%, 100=0.14%, 250=0.04%, 750=0.01%
  cpu          : usr=0.22%, sys=0.21%, ctx=51285, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=51200/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
  WRITE: io=51200MB, aggrb=38880KB/s, minb=38880KB/s, maxb=38880KB/s, mint=1348477msec, maxt=1348477msec

sudo mkdir -p /backup/fio && sudo fio randread4k.fio    && sudo rm -rf /backup/fio
randread4k: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=1
fio-2.0.13
Starting 1 process
randread4k: Laying out IO file(s) (1 file(s) / 1024MB)
Jobs: 1 (f=1): [r] [100.0% done] [8028K/0K/0K /s] [2007 /0 /0  iops] [eta 00m:00s]
randread4k: (groupid=0, jobs=1): err= 0: pid=21681: Tue Sep 23 18:50:38 2014
  read : io=5120.0MB, bw=6311.5KB/s, iops=1577 , runt=830697msec
    clat (usec): min=186 , max=28862 , avg=632.34, stdev=581.58
     lat (usec): min=186 , max=28862 , avg=632.43, stdev=581.58
    clat percentiles (usec):
     |  1.00th=[  326],  5.00th=[  330], 10.00th=[  354], 20.00th=[  358],
     | 30.00th=[  366], 40.00th=[  374], 50.00th=[  378], 60.00th=[  540],
     | 70.00th=[  548], 80.00th=[  868], 90.00th=[ 1128], 95.00th=[ 1352],
     | 99.00th=[ 2416], 99.50th=[ 3472], 99.90th=[ 8032], 99.95th=[ 9792],
     | 99.99th=[13376]
    bw (KB/s)  : min= 2328, max= 8568, per=100.00%, avg=6314.82, stdev=1118.82
    lat (usec) : 250=0.03%, 500=53.30%, 750=18.48%, 1000=14.13%
    lat (msec) : 2=12.49%, 4=1.17%, 10=0.35%, 20=0.04%, 50=0.01%
  cpu          : usr=0.35%, sys=0.54%, ctx=1492999, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=1310720/w=0/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
   READ: io=5120.0MB, aggrb=6311KB/s, minb=6311KB/s, maxb=6311KB/s, mint=830697msec, maxt=830697msec

sudo mkdir -p /backup/fio && sudo fio randwrite4k.fio   && sudo rm -rf /backup/fio
randwrite4k: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=1
fio-2.0.13
Starting 1 process
randwrite4k: Laying out IO file(s) (1 file(s) / 1024MB)
Jobs: 1 (f=1): [w] [100.0% done] [0K/3748K/0K /s] [0 /937 /0  iops] [eta 00m:00s]
randwrite4k: (groupid=0, jobs=1): err= 0: pid=21735: Tue Sep 23 19:43:41 2014
  write: io=5120.0MB, bw=2359.1KB/s, iops=589 , runt=2221582msec
    clat (usec): min=893 , max=68820 , avg=1693.12, stdev=1468.75
     lat (usec): min=893 , max=68820 , avg=1693.32, stdev=1468.89
    clat percentiles (usec):
     |  1.00th=[  956],  5.00th=[  972], 10.00th=[  980], 20.00th=[  996],
     | 30.00th=[ 1020], 40.00th=[ 1032], 50.00th=[ 1080], 60.00th=[ 1144],
     | 70.00th=[ 1336], 80.00th=[ 2800], 90.00th=[ 3152], 95.00th=[ 3856],
     | 99.00th=[ 6816], 99.50th=[ 8896], 99.90th=[17280], 99.95th=[22144],
     | 99.99th=[29824]
    bw (KB/s)  : min=  156, max= 4040, per=100.00%, avg=2363.99, stdev=1156.25
    lat (usec) : 1000=21.82%
    lat (msec) : 2=54.56%, 4=19.13%, 10=4.11%, 20=0.31%, 50=0.07%
    lat (msec) : 100=0.01%
  cpu          : usr=0.33%, sys=0.33%, ctx=1601460, majf=0, minf=22
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=1310720/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
  WRITE: io=5120.0MB, aggrb=2359KB/s, minb=2359KB/s, maxb=2359KB/s, mint=2221582msec, maxt=2221582msec

sudo umount /backup


物理サーバで構築された冗長化 NFS サーバに対する測定をします。


sudo mount -t nfs 10.110.88.62:/backup /backup

cat /proc/mounts | grep backup
10.110.88.62:/backup /backup nfs4 rw,relatime,vers=4,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.110.88.59,minorversion=0,local_lock=none,addr=10.110.88.62 0 0

df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda2             917G  3.3G  867G   1% /
tmpfs                 3.9G     0  3.9G   0% /dev/shm
/dev/sda1             248M  101M  135M  43% /boot
10.110.88.62:/backup   89G   11G   74G  13% /backup

sudo mkdir -p /backup/fio && sudo fio randread4k30.fio  && sudo rm -rf /backup/fio
randread4k30: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=1
...
randread4k30: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=1
fio-2.0.13
Starting 30 processes
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
randread4k30: Laying out IO file(s) (1 file(s) / 500MB)
Jobs: 3 (f=3): [_rrr__________________________] [100.0% done] [43040K/0K/0K /s] [10.8K/0 /0  iops] [eta 00m:00s]]
randread4k30: (groupid=0, jobs=1): err= 0: pid=22358: Wed Sep 24 20:42:41 2014
  read : io=1941.1MB, bw=3860.7KB/s, iops=965 , runt=515146msec
    clat (usec): min=374 , max=25401 , avg=1035.24, stdev=260.77
     lat (usec): min=374 , max=25401 , avg=1035.29, stdev=260.77
    clat percentiles (usec):
     |  1.00th=[  740],  5.00th=[  748], 10.00th=[  748], 20.00th=[  756],
     | 30.00th=[  764], 40.00th=[ 1112], 50.00th=[ 1112], 60.00th=[ 1128],
     | 70.00th=[ 1128], 80.00th=[ 1128], 90.00th=[ 1464], 95.00th=[ 1496],
     | 99.00th=[ 1864], 99.50th=[ 1880], 99.90th=[ 2256], 99.95th=[ 2256],
     | 99.99th=[ 2640]
    bw (KB/s)  : min= 3504, max= 4688, per=3.33%, avg=3863.65, stdev=135.18
    lat (usec) : 500=0.01%, 750=9.55%, 1000=25.55%
    lat (msec) : 2=64.75%, 4=0.15%, 50=0.01%
  cpu          : usr=0.21%, sys=0.40%, ctx=497140, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=497124/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=22359: Wed Sep 24 20:42:41 2014
  read : io=2500.0MB, bw=4114.7KB/s, iops=1028 , runt=622170msec
    clat (usec): min=271 , max=25411 , avg=971.12, stdev=276.19
     lat (usec): min=271 , max=25411 , avg=971.18, stdev=276.19
    clat percentiles (usec):
     |  1.00th=[  382],  5.00th=[  740], 10.00th=[  748], 20.00th=[  748],
     | 30.00th=[  756], 40.00th=[  756], 50.00th=[ 1112], 60.00th=[ 1112],
     | 70.00th=[ 1128], 80.00th=[ 1128], 90.00th=[ 1144], 95.00th=[ 1496],
     | 99.00th=[ 1512], 99.50th=[ 1864], 99.90th=[ 2224], 99.95th=[ 2256],
     | 99.99th=[ 2608]
    bw (KB/s)  : min= 3512, max= 8576, per=3.54%, avg=4114.70, stdev=593.52
    lat (usec) : 500=2.45%, 750=17.74%, 1000=27.32%
    lat (msec) : 2=52.37%, 4=0.11%, 50=0.01%
  cpu          : usr=0.23%, sys=0.44%, ctx=641490, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=640000/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=22360: Wed Sep 24 20:42:41 2014
  read : io=2500.0MB, bw=4115.2KB/s, iops=1028 , runt=622112msec
    clat (usec): min=267 , max=25908 , avg=971.03, stdev=277.36
     lat (usec): min=267 , max=25909 , avg=971.08, stdev=277.36
    clat percentiles (usec):
     |  1.00th=[  378],  5.00th=[  740], 10.00th=[  748], 20.00th=[  748],
     | 30.00th=[  756], 40.00th=[  756], 50.00th=[ 1112], 60.00th=[ 1112],
     | 70.00th=[ 1128], 80.00th=[ 1128], 90.00th=[ 1144], 95.00th=[ 1496],
     | 99.00th=[ 1512], 99.50th=[ 1864], 99.90th=[ 2224], 99.95th=[ 2256],
     | 99.99th=[ 2608]
    bw (KB/s)  : min= 3520, max= 9608, per=3.55%, avg=4118.14, stdev=618.41
    lat (usec) : 500=2.62%, 750=17.59%, 1000=27.18%
    lat (msec) : 2=52.49%, 4=0.12%, 50=0.01%
  cpu          : usr=0.22%, sys=0.44%, ctx=640016, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=640000/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=22361: Wed Sep 24 20:42:41 2014
  read : io=2500.0MB, bw=4115.3KB/s, iops=1028 , runt=622074msec
    clat (usec): min=261 , max=25907 , avg=970.97, stdev=276.84
     lat (usec): min=261 , max=25907 , avg=971.03, stdev=276.84
    clat percentiles (usec):
     |  1.00th=[  382],  5.00th=[  740], 10.00th=[  748], 20.00th=[  748],
     | 30.00th=[  756], 40.00th=[  756], 50.00th=[ 1112], 60.00th=[ 1112],
     | 70.00th=[ 1128], 80.00th=[ 1128], 90.00th=[ 1144], 95.00th=[ 1496],
     | 99.00th=[ 1528], 99.50th=[ 1864], 99.90th=[ 2224], 99.95th=[ 2256],
     | 99.99th=[ 2608]
    bw (KB/s)  : min= 3512, max= 8480, per=3.54%, avg=4114.31, stdev=585.61
    lat (usec) : 500=2.46%, 750=17.77%, 1000=27.35%
    lat (msec) : 2=52.30%, 4=0.12%, 10=0.01%, 50=0.01%
  cpu          : usr=0.23%, sys=0.43%, ctx=640632, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=640000/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=22362: Wed Sep 24 20:42:41 2014
  read : io=2500.0MB, bw=4117.5KB/s, iops=1029 , runt=621739msec
    clat (usec): min=291 , max=25535 , avg=970.41, stdev=277.20
     lat (usec): min=291 , max=25535 , avg=970.47, stdev=277.20
    clat percentiles (usec):
     |  1.00th=[  378],  5.00th=[  740], 10.00th=[  748], 20.00th=[  748],
     | 30.00th=[  756], 40.00th=[  756], 50.00th=[ 1112], 60.00th=[ 1112],
     | 70.00th=[ 1128], 80.00th=[ 1128], 90.00th=[ 1144], 95.00th=[ 1496],
     | 99.00th=[ 1512], 99.50th=[ 1864], 99.90th=[ 2224], 99.95th=[ 2256],
     | 99.99th=[ 2608]
    bw (KB/s)  : min= 3496, max= 7888, per=3.55%, avg=4117.42, stdev=604.62
    lat (usec) : 500=2.61%, 750=17.72%, 1000=27.21%
    lat (msec) : 2=52.36%, 4=0.11%, 10=0.01%, 50=0.01%
  cpu          : usr=0.22%, sys=0.44%, ctx=641238, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=640000/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=22363: Wed Sep 24 20:42:41 2014
  read : io=2500.0MB, bw=4118.9KB/s, iops=1029 , runt=621534msec
    clat (usec): min=302 , max=25491 , avg=970.12, stdev=275.02
     lat (usec): min=302 , max=25491 , avg=970.17, stdev=275.02
    clat percentiles (usec):
     |  1.00th=[  382],  5.00th=[  740], 10.00th=[  748], 20.00th=[  748],
     | 30.00th=[  756], 40.00th=[  756], 50.00th=[ 1112], 60.00th=[ 1112],
     | 70.00th=[ 1128], 80.00th=[ 1128], 90.00th=[ 1144], 95.00th=[ 1496],
     | 99.00th=[ 1512], 99.50th=[ 1864], 99.90th=[ 2224], 99.95th=[ 2256],
     | 99.99th=[ 2640]
    bw (KB/s)  : min= 3528, max= 7824, per=3.55%, avg=4121.78, stdev=588.44
    lat (usec) : 500=2.36%, 750=17.81%, 1000=27.55%
    lat (msec) : 2=52.16%, 4=0.11%, 10=0.01%, 50=0.01%
  cpu          : usr=0.21%, sys=0.44%, ctx=640404, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=640000/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=22364: Wed Sep 24 20:42:41 2014
  read : io=2500.0MB, bw=4144.8KB/s, iops=1036 , runt=617645msec
    clat (usec): min=334 , max=25414 , avg=964.06, stdev=263.00
     lat (usec): min=334 , max=25414 , avg=964.11, stdev=263.00
    clat percentiles (usec):
     |  1.00th=[  382],  5.00th=[  740], 10.00th=[  748], 20.00th=[  748],
     | 30.00th=[  756], 40.00th=[  756], 50.00th=[ 1112], 60.00th=[ 1112],
     | 70.00th=[ 1128], 80.00th=[ 1128], 90.00th=[ 1128], 95.00th=[ 1496],
     | 99.00th=[ 1512], 99.50th=[ 1864], 99.90th=[ 1896], 99.95th=[ 2256],
     | 99.99th=[ 2608]
    bw (KB/s)  : min= 3544, max= 7688, per=3.57%, avg=4147.66, stdev=556.45
    lat (usec) : 500=1.93%, 750=17.97%, 1000=28.17%
    lat (msec) : 2=51.85%, 4=0.08%, 10=0.01%, 50=0.01%
  cpu          : usr=0.22%, sys=0.44%, ctx=640014, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=640000/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=22365: Wed Sep 24 20:42:41 2014
  read : io=2500.0MB, bw=4161.7KB/s, iops=1040 , runt=615145msec
    clat (usec): min=335 , max=25838 , avg=960.13, stdev=255.99
     lat (usec): min=335 , max=25838 , avg=960.19, stdev=255.99
    clat percentiles (usec):
     |  1.00th=[  386],  5.00th=[  740], 10.00th=[  748], 20.00th=[  748],
     | 30.00th=[  756], 40.00th=[  756], 50.00th=[ 1112], 60.00th=[ 1112],
     | 70.00th=[ 1128], 80.00th=[ 1128], 90.00th=[ 1128], 95.00th=[ 1480],
     | 99.00th=[ 1512], 99.50th=[ 1864], 99.90th=[ 1880], 99.95th=[ 2224],
     | 99.99th=[ 2320]
    bw (KB/s)  : min= 3544, max= 6056, per=3.59%, avg=4163.71, stdev=526.68
    lat (usec) : 500=1.66%, 750=18.17%, 1000=28.64%
    lat (msec) : 2=51.46%, 4=0.06%, 10=0.01%, 50=0.01%
  cpu          : usr=0.22%, sys=0.44%, ctx=640016, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=640000/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=22366: Wed Sep 24 20:42:41 2014
  read : io=1890.8MB, bw=3914.2KB/s, iops=978 , runt=494552msec
    clat (usec): min=450 , max=25387 , avg=1020.66, stdev=245.17
     lat (usec): min=451 , max=25387 , avg=1020.72, stdev=245.17
    clat percentiles (usec):
     |  1.00th=[  740],  5.00th=[  748], 10.00th=[  748], 20.00th=[  756],
     | 30.00th=[  764], 40.00th=[ 1112], 50.00th=[ 1112], 60.00th=[ 1128],
     | 70.00th=[ 1128], 80.00th=[ 1128], 90.00th=[ 1144], 95.00th=[ 1496],
     | 99.00th=[ 1512], 99.50th=[ 1864], 99.90th=[ 1896], 99.95th=[ 2256],
     | 99.99th=[ 2384]
    bw (KB/s)  : min= 3472, max= 4432, per=3.38%, avg=3918.90, stdev=144.27
    lat (usec) : 500=0.01%, 750=9.52%, 1000=26.26%
    lat (msec) : 2=64.15%, 4=0.07%, 10=0.01%, 50=0.01%
  cpu          : usr=0.22%, sys=0.40%, ctx=484054, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=484040/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=22367: Wed Sep 24 20:42:41 2014
  read : io=2500.0MB, bw=4158.4KB/s, iops=1039 , runt=615624msec
    clat (usec): min=312 , max=26204 , avg=960.85, stdev=256.27
     lat (usec): min=312 , max=26204 , avg=960.91, stdev=256.27
    clat percentiles (usec):
     |  1.00th=[  386],  5.00th=[  740], 10.00th=[  748], 20.00th=[  748],
     | 30.00th=[  756], 40.00th=[  756], 50.00th=[ 1112], 60.00th=[ 1112],
     | 70.00th=[ 1128], 80.00th=[ 1128], 90.00th=[ 1128], 95.00th=[ 1480],
     | 99.00th=[ 1512], 99.50th=[ 1864], 99.90th=[ 1880], 99.95th=[ 2224],
     | 99.99th=[ 2352]
    bw (KB/s)  : min= 3520, max= 6144, per=3.58%, avg=4161.48, stdev=526.79
    lat (usec) : 500=1.62%, 750=18.16%, 1000=28.62%
    lat (msec) : 2=51.54%, 4=0.06%, 10=0.01%, 50=0.01%
  cpu          : usr=0.23%, sys=0.44%, ctx=640020, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=640000/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=22368: Wed Sep 24 20:42:41 2014
  read : io=2500.0MB, bw=4159.1KB/s, iops=1039 , runt=615389msec
    clat (usec): min=332 , max=25506 , avg=960.51, stdev=255.65
     lat (usec): min=332 , max=25506 , avg=960.57, stdev=255.65
    clat percentiles (usec):
     |  1.00th=[  386],  5.00th=[  740], 10.00th=[  748], 20.00th=[  748],
     | 30.00th=[  756], 40.00th=[  756], 50.00th=[ 1112], 60.00th=[ 1112],
     | 70.00th=[ 1128], 80.00th=[ 1128], 90.00th=[ 1128], 95.00th=[ 1480],
     | 99.00th=[ 1512], 99.50th=[ 1864], 99.90th=[ 1880], 99.95th=[ 2224],
     | 99.99th=[ 2352]
    bw (KB/s)  : min= 3552, max= 6016, per=3.59%, avg=4163.61, stdev=526.10
    lat (usec) : 500=1.61%, 750=18.15%, 1000=28.70%
    lat (msec) : 2=51.48%, 4=0.05%, 50=0.01%
  cpu          : usr=0.22%, sys=0.44%, ctx=640020, majf=0, minf=26
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=640000/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=22369: Wed Sep 24 20:42:41 2014
  read : io=2500.0MB, bw=4160.9KB/s, iops=1040 , runt=615255msec
    clat (usec): min=346 , max=25904 , avg=960.33, stdev=256.37
     lat (usec): min=346 , max=25904 , avg=960.39, stdev=256.37
    clat percentiles (usec):
     |  1.00th=[  386],  5.00th=[  740], 10.00th=[  748], 20.00th=[  748],
     | 30.00th=[  756], 40.00th=[  756], 50.00th=[ 1112], 60.00th=[ 1112],
     | 70.00th=[ 1128], 80.00th=[ 1128], 90.00th=[ 1128], 95.00th=[ 1480],
     | 99.00th=[ 1512], 99.50th=[ 1864], 99.90th=[ 1880], 99.95th=[ 2224],
     | 99.99th=[ 2320]
    bw (KB/s)  : min= 3544, max= 6072, per=3.59%, avg=4164.35, stdev=527.51
    lat (usec) : 500=1.64%, 750=18.08%, 1000=28.76%
    lat (msec) : 2=51.46%, 4=0.06%, 50=0.01%
  cpu          : usr=0.23%, sys=0.44%, ctx=640017, majf=0, minf=27
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=640000/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=22370: Wed Sep 24 20:42:41 2014
  read : io=2500.0MB, bw=4129.4KB/s, iops=1032 , runt=619951msec
    clat (usec): min=303 , max=25812 , avg=967.63, stdev=268.10
     lat (usec): min=303 , max=25812 , avg=967.69, stdev=268.10
    clat percentiles (usec):
     |  1.00th=[  382],  5.00th=[  740], 10.00th=[  748], 20.00th=[  748],
     | 30.00th=[  756], 40.00th=[  756], 50.00th=[ 1112], 60.00th=[ 1112],
     | 70.00th=[ 1128], 80.00th=[ 1128], 90.00th=[ 1128], 95.00th=[ 1496],
     | 99.00th=[ 1512], 99.50th=[ 1864], 99.90th=[ 1912], 99.95th=[ 2256],
     | 99.99th=[ 2608]
    bw (KB/s)  : min= 3512, max= 6800, per=3.56%, avg=4130.25, stdev=556.31
    lat (usec) : 500=2.06%, 750=17.90%, 1000=27.82%
    lat (msec) : 2=52.13%, 4=0.09%, 10=0.01%, 50=0.01%
  cpu          : usr=0.22%, sys=0.44%, ctx=640018, majf=0, minf=26
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=640000/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=22371: Wed Sep 24 20:42:41 2014
  read : io=1978.4MB, bw=3889.8KB/s, iops=972 , runt=520818msec
    clat (usec): min=353 , max=26284 , avg=1027.27, stdev=261.84
     lat (usec): min=353 , max=26284 , avg=1027.33, stdev=261.84
    clat percentiles (usec):
     |  1.00th=[  740],  5.00th=[  748], 10.00th=[  748], 20.00th=[  756],
     | 30.00th=[  756], 40.00th=[ 1112], 50.00th=[ 1112], 60.00th=[ 1128],
     | 70.00th=[ 1128], 80.00th=[ 1128], 90.00th=[ 1144], 95.00th=[ 1496],
     | 99.00th=[ 1864], 99.50th=[ 1880], 99.90th=[ 2256], 99.95th=[ 2256],
     | 99.99th=[ 2640]
    bw (KB/s)  : min= 3512, max= 5248, per=3.35%, avg=3892.64, stdev=172.25
    lat (usec) : 500=0.08%, 750=10.36%, 1000=26.39%
    lat (msec) : 2=63.04%, 4=0.14%, 50=0.01%
  cpu          : usr=0.21%, sys=0.41%, ctx=506481, majf=0, minf=26
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=506464/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=22372: Wed Sep 24 20:42:41 2014
  read : io=2500.0MB, bw=4120.4KB/s, iops=1030 , runt=621306msec
    clat (usec): min=323 , max=25917 , avg=969.76, stdev=276.65
     lat (usec): min=323 , max=25917 , avg=969.82, stdev=276.65
    clat percentiles (usec):
     |  1.00th=[  378],  5.00th=[  740], 10.00th=[  748], 20.00th=[  748],
     | 30.00th=[  756], 40.00th=[  756], 50.00th=[ 1112], 60.00th=[ 1112],
     | 70.00th=[ 1128], 80.00th=[ 1128], 90.00th=[ 1144], 95.00th=[ 1496],
     | 99.00th=[ 1512], 99.50th=[ 1864], 99.90th=[ 2224], 99.95th=[ 2256],
     | 99.99th=[ 2640]
    bw (KB/s)  : min= 3568, max= 8872, per=3.55%, avg=4122.89, stdev=607.97
    lat (usec) : 500=2.57%, 750=17.58%, 1000=27.54%
    lat (msec) : 2=52.19%, 4=0.12%, 10=0.01%, 50=0.01%
  cpu          : usr=0.22%, sys=0.44%, ctx=640020, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=640000/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=22373: Wed Sep 24 20:42:41 2014
  read : io=1949.9MB, bw=3867.2KB/s, iops=966 , runt=516312msec
    clat (usec): min=363 , max=25907 , avg=1033.34, stdev=260.62
     lat (usec): min=363 , max=25907 , avg=1033.40, stdev=260.62
    clat percentiles (usec):
     |  1.00th=[  740],  5.00th=[  748], 10.00th=[  748], 20.00th=[  756],
     | 30.00th=[  764], 40.00th=[ 1112], 50.00th=[ 1112], 60.00th=[ 1128],
     | 70.00th=[ 1128], 80.00th=[ 1128], 90.00th=[ 1160], 95.00th=[ 1496],
     | 99.00th=[ 1864], 99.50th=[ 1880], 99.90th=[ 2256], 99.95th=[ 2256],
     | 99.99th=[ 2640]
    bw (KB/s)  : min= 3441, max= 4800, per=3.33%, avg=3870.14, stdev=126.70
    lat (usec) : 500=0.01%, 750=9.53%, 1000=25.92%
    lat (msec) : 2=64.39%, 4=0.14%, 50=0.01%
  cpu          : usr=0.21%, sys=0.40%, ctx=499174, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=499160/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=22374: Wed Sep 24 20:42:41 2014
  read : io=2500.0MB, bw=4120.3KB/s, iops=1030 , runt=621324msec
    clat (usec): min=319 , max=25542 , avg=969.79, stdev=276.04
     lat (usec): min=319 , max=25542 , avg=969.85, stdev=276.04
    clat percentiles (usec):
     |  1.00th=[  382],  5.00th=[  740], 10.00th=[  748], 20.00th=[  748],
     | 30.00th=[  756], 40.00th=[  756], 50.00th=[ 1112], 60.00th=[ 1112],
     | 70.00th=[ 1128], 80.00th=[ 1128], 90.00th=[ 1144], 95.00th=[ 1496],
     | 99.00th=[ 1512], 99.50th=[ 1864], 99.90th=[ 2224], 99.95th=[ 2256],
     | 99.99th=[ 2608]
    bw (KB/s)  : min= 3592, max= 7736, per=3.55%, avg=4121.31, stdev=587.69
    lat (usec) : 500=2.43%, 750=17.71%, 1000=27.68%
    lat (msec) : 2=52.07%, 4=0.11%, 50=0.01%
  cpu          : usr=0.22%, sys=0.44%, ctx=640496, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=640000/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=22375: Wed Sep 24 20:42:41 2014
  read : io=2500.0MB, bw=4120.5KB/s, iops=1030 , runt=621300msec
    clat (usec): min=331 , max=25916 , avg=969.74, stdev=275.52
     lat (usec): min=331 , max=25916 , avg=969.80, stdev=275.52
    clat percentiles (usec):
     |  1.00th=[  382],  5.00th=[  740], 10.00th=[  748], 20.00th=[  748],
     | 30.00th=[  756], 40.00th=[  756], 50.00th=[ 1112], 60.00th=[ 1112],
     | 70.00th=[ 1128], 80.00th=[ 1128], 90.00th=[ 1144], 95.00th=[ 1496],
     | 99.00th=[ 1512], 99.50th=[ 1864], 99.90th=[ 2224], 99.95th=[ 2256],
     | 99.99th=[ 2608]
    bw (KB/s)  : min= 3576, max= 8248, per=3.55%, avg=4121.80, stdev=586.91
    lat (usec) : 500=2.42%, 750=17.78%, 1000=27.53%
    lat (msec) : 2=52.14%, 4=0.12%, 50=0.01%
  cpu          : usr=0.22%, sys=0.44%, ctx=640018, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=640000/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=22376: Wed Sep 24 20:42:41 2014
  read : io=2500.0MB, bw=4134.6KB/s, iops=1033 , runt=619177msec
    clat (usec): min=327 , max=25378 , avg=966.42, stdev=270.20
     lat (usec): min=327 , max=25378 , avg=966.48, stdev=270.20
    clat percentiles (usec):
     |  1.00th=[  382],  5.00th=[  740], 10.00th=[  748], 20.00th=[  748],
     | 30.00th=[  756], 40.00th=[  756], 50.00th=[ 1112], 60.00th=[ 1112],
     | 70.00th=[ 1128], 80.00th=[ 1128], 90.00th=[ 1128], 95.00th=[ 1496],
     | 99.00th=[ 1512], 99.50th=[ 1864], 99.90th=[ 1928], 99.95th=[ 2256],
     | 99.99th=[ 2608]
    bw (KB/s)  : min= 3616, max= 7552, per=3.56%, avg=4135.51, stdev=555.92
    lat (usec) : 500=2.10%, 750=18.11%, 1000=28.12%
    lat (msec) : 2=51.58%, 4=0.09%, 10=0.01%, 50=0.01%
  cpu          : usr=0.22%, sys=0.44%, ctx=640019, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=640000/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=22377: Wed Sep 24 20:42:41 2014
  read : io=2500.0MB, bw=4171.3KB/s, iops=1042 , runt=613722msec
    clat (usec): min=341 , max=25540 , avg=957.93, stdev=254.99
     lat (usec): min=341 , max=25540 , avg=957.98, stdev=254.99
    clat percentiles (usec):
     |  1.00th=[  386],  5.00th=[  740], 10.00th=[  748], 20.00th=[  748],
     | 30.00th=[  756], 40.00th=[  756], 50.00th=[ 1096], 60.00th=[ 1112],
     | 70.00th=[ 1128], 80.00th=[ 1128], 90.00th=[ 1128], 95.00th=[ 1480],
     | 99.00th=[ 1512], 99.50th=[ 1848], 99.90th=[ 1880], 99.95th=[ 2224],
     | 99.99th=[ 2352]
    bw (KB/s)  : min= 3648, max= 5856, per=3.60%, avg=4174.65, stdev=521.56
    lat (usec) : 500=1.61%, 750=18.14%, 1000=29.29%
    lat (msec) : 2=50.92%, 4=0.06%, 50=0.01%
  cpu          : usr=0.23%, sys=0.44%, ctx=640015, majf=0, minf=26
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=640000/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=22378: Wed Sep 24 20:42:41 2014
  read : io=1996.2MB, bw=3959.4KB/s, iops=989 , runt=516246msec
    clat (usec): min=356 , max=25552 , avg=1009.22, stdev=245.35
     lat (usec): min=356 , max=25552 , avg=1009.28, stdev=245.35
    clat percentiles (usec):
     |  1.00th=[  740],  5.00th=[  748], 10.00th=[  748], 20.00th=[  756],
     | 30.00th=[  756], 40.00th=[ 1112], 50.00th=[ 1112], 60.00th=[ 1112],
     | 70.00th=[ 1128], 80.00th=[ 1128], 90.00th=[ 1128], 95.00th=[ 1496],
     | 99.00th=[ 1512], 99.50th=[ 1864], 99.90th=[ 1896], 99.95th=[ 2256],
     | 99.99th=[ 2608]
    bw (KB/s)  : min= 3608, max= 4856, per=3.41%, avg=3962.57, stdev=157.50
    lat (usec) : 500=0.01%, 750=10.50%, 1000=27.92%
    lat (msec) : 2=61.50%, 4=0.07%, 50=0.01%
  cpu          : usr=0.22%, sys=0.41%, ctx=511019, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=511004/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=22379: Wed Sep 24 20:42:41 2014
  read : io=1880.6MB, bw=3938.6KB/s, iops=984 , runt=488931msec
    clat (usec): min=444 , max=25533 , avg=1014.56, stdev=246.28
     lat (usec): min=444 , max=25533 , avg=1014.62, stdev=246.28
    clat percentiles (usec):
     |  1.00th=[  740],  5.00th=[  748], 10.00th=[  748], 20.00th=[  756],
     | 30.00th=[  756], 40.00th=[ 1112], 50.00th=[ 1112], 60.00th=[ 1128],
     | 70.00th=[ 1128], 80.00th=[ 1128], 90.00th=[ 1128], 95.00th=[ 1496],
     | 99.00th=[ 1512], 99.50th=[ 1864], 99.90th=[ 1896], 99.95th=[ 2256],
     | 99.99th=[ 2608]
    bw (KB/s)  : min= 3488, max= 4576, per=3.40%, avg=3942.39, stdev=176.25
    lat (usec) : 500=0.01%, 750=9.95%, 1000=27.33%
    lat (msec) : 2=62.65%, 4=0.07%, 50=0.01%
  cpu          : usr=0.20%, sys=0.42%, ctx=481444, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=481416/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=22380: Wed Sep 24 20:42:41 2014
  read : io=2500.0MB, bw=4179.4KB/s, iops=1044 , runt=612533msec
    clat (usec): min=327 , max=25367 , avg=956.04, stdev=254.97
     lat (usec): min=327 , max=25367 , avg=956.09, stdev=254.97
    clat percentiles (usec):
     |  1.00th=[  386],  5.00th=[  740], 10.00th=[  748], 20.00th=[  748],
     | 30.00th=[  756], 40.00th=[  756], 50.00th=[ 1096], 60.00th=[ 1112],
     | 70.00th=[ 1128], 80.00th=[ 1128], 90.00th=[ 1128], 95.00th=[ 1480],
     | 99.00th=[ 1512], 99.50th=[ 1864], 99.90th=[ 1880], 99.95th=[ 2224],
     | 99.99th=[ 2320]
    bw (KB/s)  : min= 3520, max= 5984, per=3.60%, avg=4183.03, stdev=516.82
    lat (usec) : 500=1.53%, 750=18.30%, 1000=29.76%
    lat (msec) : 2=50.35%, 4=0.05%, 10=0.01%, 50=0.01%
  cpu          : usr=0.22%, sys=0.44%, ctx=640022, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=640000/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=22381: Wed Sep 24 20:42:41 2014
  read : io=2500.0MB, bw=4181.6KB/s, iops=1045 , runt=612219msec
    clat (usec): min=331 , max=25531 , avg=955.57, stdev=255.13
     lat (usec): min=331 , max=25531 , avg=955.62, stdev=255.13
    clat percentiles (usec):
     |  1.00th=[  386],  5.00th=[  740], 10.00th=[  748], 20.00th=[  748],
     | 30.00th=[  756], 40.00th=[  756], 50.00th=[ 1096], 60.00th=[ 1112],
     | 70.00th=[ 1128], 80.00th=[ 1128], 90.00th=[ 1128], 95.00th=[ 1480],
     | 99.00th=[ 1512], 99.50th=[ 1864], 99.90th=[ 1880], 99.95th=[ 2224],
     | 99.99th=[ 2576]
    bw (KB/s)  : min= 3528, max= 5680, per=3.60%, avg=4184.57, stdev=519.64
    lat (usec) : 500=1.58%, 750=18.23%, 1000=29.87%
    lat (msec) : 2=50.25%, 4=0.06%, 50=0.01%
  cpu          : usr=0.22%, sys=0.44%, ctx=640024, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=640000/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=22382: Wed Sep 24 20:42:41 2014
  read : io=2500.0MB, bw=4171.7KB/s, iops=1042 , runt=613671msec
    clat (usec): min=349 , max=25915 , avg=957.83, stdev=257.78
     lat (usec): min=349 , max=25915 , avg=957.88, stdev=257.78
    clat percentiles (usec):
     |  1.00th=[  386],  5.00th=[  740], 10.00th=[  748], 20.00th=[  748],
     | 30.00th=[  756], 40.00th=[  756], 50.00th=[ 1096], 60.00th=[ 1112],
     | 70.00th=[ 1128], 80.00th=[ 1128], 90.00th=[ 1128], 95.00th=[ 1496],
     | 99.00th=[ 1512], 99.50th=[ 1864], 99.90th=[ 1880], 99.95th=[ 2256],
     | 99.99th=[ 2352]
    bw (KB/s)  : min= 3496, max= 5896, per=3.60%, avg=4175.26, stdev=517.37
    lat (usec) : 500=1.52%, 750=18.26%, 1000=29.80%
    lat (msec) : 2=50.35%, 4=0.07%, 50=0.01%
  cpu          : usr=0.22%, sys=0.44%, ctx=640019, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=640000/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=22383: Wed Sep 24 20:42:41 2014
  read : io=2500.0MB, bw=4144.6KB/s, iops=1036 , runt=617685msec
    clat (usec): min=319 , max=25898 , avg=964.09, stdev=271.09
     lat (usec): min=319 , max=25898 , avg=964.15, stdev=271.09
    clat percentiles (usec):
     |  1.00th=[  386],  5.00th=[  740], 10.00th=[  748], 20.00th=[  748],
     | 30.00th=[  756], 40.00th=[  756], 50.00th=[ 1096], 60.00th=[ 1112],
     | 70.00th=[ 1128], 80.00th=[ 1128], 90.00th=[ 1128], 95.00th=[ 1496],
     | 99.00th=[ 1512], 99.50th=[ 1864], 99.90th=[ 2224], 99.95th=[ 2256],
     | 99.99th=[ 2608]
    bw (KB/s)  : min= 3448, max= 6672, per=3.57%, avg=4147.15, stdev=543.24
    lat (usec) : 500=1.86%, 750=18.41%, 1000=29.17%
    lat (msec) : 2=50.44%, 4=0.11%, 50=0.01%
  cpu          : usr=0.22%, sys=0.44%, ctx=640019, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=640000/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=22384: Wed Sep 24 20:42:41 2014
  read : io=2500.0MB, bw=4135.6KB/s, iops=1033 , runt=619029msec
    clat (usec): min=340 , max=25768 , avg=966.18, stdev=271.46
     lat (usec): min=340 , max=25768 , avg=966.24, stdev=271.46
    clat percentiles (usec):
     |  1.00th=[  386],  5.00th=[  740], 10.00th=[  748], 20.00th=[  748],
     | 30.00th=[  756], 40.00th=[  756], 50.00th=[ 1096], 60.00th=[ 1112],
     | 70.00th=[ 1128], 80.00th=[ 1128], 90.00th=[ 1128], 95.00th=[ 1496],
     | 99.00th=[ 1512], 99.50th=[ 1864], 99.90th=[ 2224], 99.95th=[ 2256],
     | 99.99th=[ 2608]
    bw (KB/s)  : min= 3456, max= 6424, per=3.56%, avg=4138.16, stdev=544.36
    lat (usec) : 500=1.87%, 750=18.17%, 1000=28.96%
    lat (msec) : 2=50.89%, 4=0.11%, 50=0.01%
  cpu          : usr=0.23%, sys=0.44%, ctx=640019, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=640000/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=22385: Wed Sep 24 20:42:41 2014
  read : io=1941.1MB, bw=3903.5KB/s, iops=975 , runt=509429msec
    clat (usec): min=336 , max=25406 , avg=1023.72, stdev=262.24
     lat (usec): min=336 , max=25406 , avg=1023.77, stdev=262.24
    clat percentiles (usec):
     |  1.00th=[  740],  5.00th=[  748], 10.00th=[  748], 20.00th=[  756],
     | 30.00th=[  756], 40.00th=[ 1112], 50.00th=[ 1112], 60.00th=[ 1128],
     | 70.00th=[ 1128], 80.00th=[ 1128], 90.00th=[ 1144], 95.00th=[ 1496],
     | 99.00th=[ 1864], 99.50th=[ 1880], 99.90th=[ 2256], 99.95th=[ 2256],
     | 99.99th=[ 2608]
    bw (KB/s)  : min= 3392, max= 4664, per=3.37%, avg=3906.95, stdev=212.91
    lat (usec) : 500=0.01%, 750=10.15%, 1000=27.57%
    lat (msec) : 2=62.14%, 4=0.14%, 50=0.01%
  cpu          : usr=0.21%, sys=0.41%, ctx=497143, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=497128/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=22386: Wed Sep 24 20:42:41 2014
  read : io=2500.0MB, bw=4140.2KB/s, iops=1035 , runt=618342msec
    clat (usec): min=295 , max=25915 , avg=965.14, stdev=270.65
     lat (usec): min=295 , max=25915 , avg=965.19, stdev=270.65
    clat percentiles (usec):
     |  1.00th=[  386],  5.00th=[  740], 10.00th=[  748], 20.00th=[  748],
     | 30.00th=[  756], 40.00th=[  756], 50.00th=[ 1096], 60.00th=[ 1112],
     | 70.00th=[ 1128], 80.00th=[ 1128], 90.00th=[ 1128], 95.00th=[ 1496],
     | 99.00th=[ 1512], 99.50th=[ 1864], 99.90th=[ 2224], 99.95th=[ 2256],
     | 99.99th=[ 2608]
    bw (KB/s)  : min= 3464, max= 6600, per=3.57%, avg=4142.43, stdev=544.73
    lat (usec) : 500=1.82%, 750=18.29%, 1000=29.14%
    lat (msec) : 2=50.64%, 4=0.11%, 10=0.01%, 50=0.01%
  cpu          : usr=0.22%, sys=0.45%, ctx=640022, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=640000/w=0/d=0, short=r=0/w=0/d=0
randread4k30: (groupid=0, jobs=1): err= 0: pid=22387: Wed Sep 24 20:42:41 2014
  read : io=1961.1MB, bw=3908.9KB/s, iops=977 , runt=513964msec
    clat (usec): min=378 , max=25498 , avg=1022.32, stdev=262.25
     lat (usec): min=378 , max=25498 , avg=1022.37, stdev=262.25
    clat percentiles (usec):
     |  1.00th=[  740],  5.00th=[  748], 10.00th=[  748], 20.00th=[  756],
     | 30.00th=[  756], 40.00th=[ 1112], 50.00th=[ 1112], 60.00th=[ 1128],
     | 70.00th=[ 1128], 80.00th=[ 1128], 90.00th=[ 1144], 95.00th=[ 1496],
     | 99.00th=[ 1864], 99.50th=[ 1880], 99.90th=[ 2256], 99.95th=[ 2256],
     | 99.99th=[ 2640]
    bw (KB/s)  : min= 3400, max= 4856, per=3.37%, avg=3912.42, stdev=208.65
    lat (usec) : 500=0.01%, 750=10.33%, 1000=27.67%
    lat (msec) : 2=61.84%, 4=0.15%, 50=0.01%
  cpu          : usr=0.21%, sys=0.40%, ctx=502575, majf=0, minf=26
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=502244/w=0/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
   READ: io=70541MB, aggrb=116100KB/s, minb=3860KB/s, maxb=4181KB/s, mint=488931msec, maxt=622170msec

sudo mkdir -p /backup/fio && sudo fio randwrite4k30.fio && sudo rm -rf /backup/fio
randwrite4k30: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=1
...
randwrite4k30: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=1
fio-2.0.13
Starting 30 processes
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
randwrite4k30: Laying out IO file(s) (1 file(s) / 500MB)
Jobs: 5 (f=5): [_w___w_w___________w___w______] [99.9% done] [0K/77624K/0K /s] [0 /19.5K/0  iops] [eta 00m:01s]s]
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=22402: Wed Sep 24 20:59:53 2014
  write: io=1941.1MB, bw=2261.7KB/s, iops=565 , runt=879234msec
    clat (usec): min=394 , max=111911 , avg=1767.44, stdev=1610.36
     lat (usec): min=394 , max=111911 , avg=1767.61, stdev=1610.35
    clat percentiles (usec):
     |  1.00th=[  716],  5.00th=[  756], 10.00th=[  836], 20.00th=[ 1020],
     | 30.00th=[ 1064], 40.00th=[ 1096], 50.00th=[ 1128], 60.00th=[ 1144],
     | 70.00th=[ 1352], 80.00th=[ 2736], 90.00th=[ 3312], 95.00th=[ 4384],
     | 99.00th=[ 7584], 99.50th=[ 9408], 99.90th=[18048], 99.95th=[21888],
     | 99.99th=[28544]
    bw (KB/s)  : min=  552, max= 3992, per=3.21%, avg=2265.10, stdev=1303.09
    lat (usec) : 500=0.01%, 750=3.30%, 1000=14.33%
    lat (msec) : 2=55.93%, 4=20.31%, 10=5.71%, 20=0.33%, 50=0.07%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=0.14%, sys=0.22%, ctx=498035, majf=0, minf=22
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=497124/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=22403: Wed Sep 24 20:59:53 2014
  write: io=2500.0MB, bw=2501.1KB/s, iops=625 , runt=1023205msec
    clat (usec): min=358 , max=113445 , avg=1597.52, stdev=1487.14
     lat (usec): min=358 , max=113445 , avg=1597.68, stdev=1487.14
    clat percentiles (usec):
     |  1.00th=[  668],  5.00th=[  740], 10.00th=[  764], 20.00th=[  956],
     | 30.00th=[ 1032], 40.00th=[ 1080], 50.00th=[ 1096], 60.00th=[ 1128],
     | 70.00th=[ 1176], 80.00th=[ 2320], 90.00th=[ 3120], 95.00th=[ 3952],
     | 99.00th=[ 7136], 99.50th=[ 8896], 99.90th=[17024], 99.95th=[21376],
     | 99.99th=[28032]
    bw (KB/s)  : min=  550, max= 4296, per=3.55%, avg=2505.15, stdev=1354.36
    lat (usec) : 500=0.03%, 750=6.22%, 1000=18.51%
    lat (msec) : 2=54.43%, 4=15.94%, 10=4.53%, 20=0.28%, 50=0.06%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=0.15%, sys=0.24%, ctx=641424, majf=0, minf=22
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=640000/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=22404: Wed Sep 24 20:59:53 2014
  write: io=2500.0MB, bw=2502.4KB/s, iops=625 , runt=1023031msec
    clat (usec): min=362 , max=102256 , avg=1597.31, stdev=1486.74
     lat (usec): min=363 , max=102256 , avg=1597.47, stdev=1486.74
    clat percentiles (usec):
     |  1.00th=[  668],  5.00th=[  740], 10.00th=[  772], 20.00th=[  956],
     | 30.00th=[ 1032], 40.00th=[ 1080], 50.00th=[ 1096], 60.00th=[ 1128],
     | 70.00th=[ 1176], 80.00th=[ 2320], 90.00th=[ 3120], 95.00th=[ 3952],
     | 99.00th=[ 7136], 99.50th=[ 8896], 99.90th=[17024], 99.95th=[21376],
     | 99.99th=[27520]
    bw (KB/s)  : min=  540, max= 4320, per=3.55%, avg=2506.03, stdev=1354.02
    lat (usec) : 500=0.03%, 750=6.24%, 1000=18.43%
    lat (msec) : 2=54.48%, 4=15.97%, 10=4.51%, 20=0.28%, 50=0.06%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=0.17%, sys=0.23%, ctx=641868, majf=0, minf=22
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=640000/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=22405: Wed Sep 24 20:59:53 2014
  write: io=2500.0MB, bw=2502.8KB/s, iops=625 , runt=1022871msec
    clat (usec): min=346 , max=92618 , avg=1597.05, stdev=1482.23
     lat (usec): min=346 , max=92618 , avg=1597.21, stdev=1482.22
    clat percentiles (usec):
     |  1.00th=[  668],  5.00th=[  740], 10.00th=[  764], 20.00th=[  956],
     | 30.00th=[ 1032], 40.00th=[ 1080], 50.00th=[ 1096], 60.00th=[ 1128],
     | 70.00th=[ 1176], 80.00th=[ 2320], 90.00th=[ 3120], 95.00th=[ 3952],
     | 99.00th=[ 7136], 99.50th=[ 8896], 99.90th=[17024], 99.95th=[21376],
     | 99.99th=[27008]
    bw (KB/s)  : min=  536, max= 4311, per=3.55%, avg=2506.34, stdev=1354.08
    lat (usec) : 500=0.03%, 750=6.25%, 1000=18.48%
    lat (msec) : 2=54.41%, 4=15.98%, 10=4.51%, 20=0.28%, 50=0.06%
    lat (msec) : 100=0.01%
  cpu          : usr=0.16%, sys=0.23%, ctx=640777, majf=0, minf=22
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=640000/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=22406: Wed Sep 24 20:59:53 2014
  write: io=2500.0MB, bw=2502.3KB/s, iops=625 , runt=1023070msec
    clat (usec): min=350 , max=112282 , avg=1597.34, stdev=1483.89
     lat (usec): min=351 , max=112282 , avg=1597.50, stdev=1483.88
    clat percentiles (usec):
     |  1.00th=[  668],  5.00th=[  740], 10.00th=[  764], 20.00th=[  956],
     | 30.00th=[ 1032], 40.00th=[ 1080], 50.00th=[ 1096], 60.00th=[ 1128],
     | 70.00th=[ 1176], 80.00th=[ 2320], 90.00th=[ 3120], 95.00th=[ 3952],
     | 99.00th=[ 7136], 99.50th=[ 8896], 99.90th=[17024], 99.95th=[21120],
     | 99.99th=[27264]
    bw (KB/s)  : min=  543, max= 4288, per=3.55%, avg=2506.07, stdev=1353.28
    lat (usec) : 500=0.03%, 750=6.25%, 1000=18.41%
    lat (msec) : 2=54.47%, 4=15.99%, 10=4.49%, 20=0.29%, 50=0.06%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=0.16%, sys=0.23%, ctx=640604, majf=0, minf=22
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=640000/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=22407: Wed Sep 24 20:59:53 2014
  write: io=2500.0MB, bw=2501.1KB/s, iops=625 , runt=1023219msec
    clat (usec): min=360 , max=106139 , avg=1597.58, stdev=1486.73
     lat (usec): min=361 , max=106139 , avg=1597.74, stdev=1486.72
    clat percentiles (usec):
     |  1.00th=[  668],  5.00th=[  740], 10.00th=[  772], 20.00th=[  956],
     | 30.00th=[ 1032], 40.00th=[ 1080], 50.00th=[ 1096], 60.00th=[ 1128],
     | 70.00th=[ 1176], 80.00th=[ 2320], 90.00th=[ 3120], 95.00th=[ 3952],
     | 99.00th=[ 7136], 99.50th=[ 8896], 99.90th=[16768], 99.95th=[21376],
     | 99.99th=[28288]
    bw (KB/s)  : min=  538, max= 4296, per=3.55%, avg=2505.31, stdev=1354.54
    lat (usec) : 500=0.04%, 750=6.23%, 1000=18.47%
    lat (msec) : 2=54.47%, 4=15.93%, 10=4.52%, 20=0.29%, 50=0.06%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=0.17%, sys=0.23%, ctx=640861, majf=0, minf=24
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=640000/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=22408: Wed Sep 24 20:59:53 2014
  write: io=2500.0MB, bw=2502.6KB/s, iops=625 , runt=1022947msec
    clat (usec): min=364 , max=94179 , avg=1597.15, stdev=1481.71
     lat (usec): min=364 , max=94179 , avg=1597.31, stdev=1481.71
    clat percentiles (usec):
     |  1.00th=[  668],  5.00th=[  740], 10.00th=[  764], 20.00th=[  956],
     | 30.00th=[ 1032], 40.00th=[ 1080], 50.00th=[ 1096], 60.00th=[ 1128],
     | 70.00th=[ 1176], 80.00th=[ 2320], 90.00th=[ 3120], 95.00th=[ 3952],
     | 99.00th=[ 7136], 99.50th=[ 8896], 99.90th=[16768], 99.95th=[21376],
     | 99.99th=[27776]
    bw (KB/s)  : min=  540, max= 4328, per=3.55%, avg=2505.42, stdev=1354.80
    lat (usec) : 500=0.03%, 750=6.21%, 1000=18.53%
    lat (msec) : 2=54.41%, 4=15.95%, 10=4.53%, 20=0.28%, 50=0.06%
    lat (msec) : 100=0.01%
  cpu          : usr=0.16%, sys=0.23%, ctx=641162, majf=0, minf=22
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=640000/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=22409: Wed Sep 24 20:59:53 2014
  write: io=2500.0MB, bw=2501.4KB/s, iops=625 , runt=1023451msec
    clat (usec): min=347 , max=108005 , avg=1597.96, stdev=1485.05
     lat (usec): min=347 , max=108005 , avg=1598.12, stdev=1485.04
    clat percentiles (usec):
     |  1.00th=[  668],  5.00th=[  740], 10.00th=[  764], 20.00th=[  956],
     | 30.00th=[ 1032], 40.00th=[ 1080], 50.00th=[ 1096], 60.00th=[ 1128],
     | 70.00th=[ 1176], 80.00th=[ 2320], 90.00th=[ 3120], 95.00th=[ 3952],
     | 99.00th=[ 7136], 99.50th=[ 8896], 99.90th=[17024], 99.95th=[21376],
     | 99.99th=[27264]
    bw (KB/s)  : min=  536, max= 4280, per=3.55%, avg=2504.48, stdev=1353.81
    lat (usec) : 500=0.03%, 750=6.22%, 1000=18.46%
    lat (msec) : 2=54.48%, 4=15.94%, 10=4.52%, 20=0.28%, 50=0.06%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=0.16%, sys=0.23%, ctx=640879, majf=0, minf=22
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=640000/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=22410: Wed Sep 24 20:59:53 2014
  write: io=1890.8MB, bw=2239.6KB/s, iops=559 , runt=864519msec
    clat (usec): min=401 , max=106125 , avg=1784.82, stdev=1615.63
     lat (usec): min=401 , max=106125 , avg=1784.98, stdev=1615.62
    clat percentiles (usec):
     |  1.00th=[  716],  5.00th=[  764], 10.00th=[  852], 20.00th=[ 1020],
     | 30.00th=[ 1064], 40.00th=[ 1096], 50.00th=[ 1128], 60.00th=[ 1144],
     | 70.00th=[ 1368], 80.00th=[ 2768], 90.00th=[ 3312], 95.00th=[ 4448],
     | 99.00th=[ 7648], 99.50th=[ 9536], 99.90th=[18048], 99.95th=[21888],
     | 99.99th=[28288]
    bw (KB/s)  : min=  536, max= 3896, per=3.18%, avg=2243.09, stdev=1299.37
    lat (usec) : 500=0.01%, 750=3.14%, 1000=14.10%
    lat (msec) : 2=55.62%, 4=20.86%, 10=5.86%, 20=0.33%, 50=0.08%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=0.15%, sys=0.21%, ctx=486506, majf=0, minf=23
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=484040/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=22411: Wed Sep 24 20:59:53 2014
  write: io=2500.0MB, bw=2502.6KB/s, iops=625 , runt=1022978msec
    clat (usec): min=359 , max=114226 , avg=1597.21, stdev=1486.20
     lat (usec): min=359 , max=114226 , avg=1597.38, stdev=1486.19
    clat percentiles (usec):
     |  1.00th=[  668],  5.00th=[  740], 10.00th=[  772], 20.00th=[  956],
     | 30.00th=[ 1032], 40.00th=[ 1080], 50.00th=[ 1096], 60.00th=[ 1128],
     | 70.00th=[ 1176], 80.00th=[ 2320], 90.00th=[ 3120], 95.00th=[ 3952],
     | 99.00th=[ 7136], 99.50th=[ 8896], 99.90th=[17024], 99.95th=[21376],
     | 99.99th=[27520]
    bw (KB/s)  : min=  526, max= 4296, per=3.55%, avg=2506.57, stdev=1353.97
    lat (usec) : 500=0.03%, 750=6.24%, 1000=18.46%
    lat (msec) : 2=54.43%, 4=15.98%, 10=4.51%, 20=0.28%, 50=0.06%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=0.16%, sys=0.23%, ctx=640476, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=640000/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=22412: Wed Sep 24 20:59:53 2014
  write: io=2500.0MB, bw=2502.6KB/s, iops=625 , runt=1022952msec
    clat (usec): min=368 , max=112296 , avg=1597.16, stdev=1483.09
     lat (usec): min=368 , max=112296 , avg=1597.32, stdev=1483.09
    clat percentiles (usec):
     |  1.00th=[  668],  5.00th=[  740], 10.00th=[  764], 20.00th=[  956],
     | 30.00th=[ 1032], 40.00th=[ 1080], 50.00th=[ 1096], 60.00th=[ 1128],
     | 70.00th=[ 1176], 80.00th=[ 2320], 90.00th=[ 3120], 95.00th=[ 3952],
     | 99.00th=[ 7136], 99.50th=[ 8896], 99.90th=[17024], 99.95th=[21376],
     | 99.99th=[27264]
    bw (KB/s)  : min=  544, max= 4328, per=3.55%, avg=2505.74, stdev=1354.18
    lat (usec) : 500=0.03%, 750=6.25%, 1000=18.52%
    lat (msec) : 2=54.38%, 4=15.98%, 10=4.50%, 20=0.28%, 50=0.06%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=0.16%, sys=0.24%, ctx=641079, majf=0, minf=24
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=640000/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=22413: Wed Sep 24 20:59:53 2014
  write: io=2500.0MB, bw=2502.9KB/s, iops=625 , runt=1022839msec
    clat (usec): min=369 , max=113445 , avg=1596.98, stdev=1481.95
     lat (usec): min=369 , max=113445 , avg=1597.14, stdev=1481.95
    clat percentiles (usec):
     |  1.00th=[  668],  5.00th=[  740], 10.00th=[  764], 20.00th=[  956],
     | 30.00th=[ 1032], 40.00th=[ 1080], 50.00th=[ 1096], 60.00th=[ 1128],
     | 70.00th=[ 1176], 80.00th=[ 2320], 90.00th=[ 3120], 95.00th=[ 3952],
     | 99.00th=[ 7136], 99.50th=[ 8896], 99.90th=[17024], 99.95th=[21376],
     | 99.99th=[27776]
    bw (KB/s)  : min=  549, max= 4320, per=3.55%, avg=2506.52, stdev=1352.23
    lat (usec) : 500=0.03%, 750=6.20%, 1000=18.52%
    lat (msec) : 2=54.40%, 4=15.99%, 10=4.51%, 20=0.28%, 50=0.06%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=0.16%, sys=0.23%, ctx=641560, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=640000/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=22414: Wed Sep 24 20:59:53 2014
  write: io=2500.0MB, bw=2502.7KB/s, iops=625 , runt=1022913msec
    clat (usec): min=357 , max=111915 , avg=1597.10, stdev=1484.42
     lat (usec): min=358 , max=111915 , avg=1597.26, stdev=1484.42
    clat percentiles (usec):
     |  1.00th=[  668],  5.00th=[  740], 10.00th=[  764], 20.00th=[  956],
     | 30.00th=[ 1032], 40.00th=[ 1080], 50.00th=[ 1096], 60.00th=[ 1128],
     | 70.00th=[ 1176], 80.00th=[ 2320], 90.00th=[ 3120], 95.00th=[ 3952],
     | 99.00th=[ 7136], 99.50th=[ 8896], 99.90th=[17024], 99.95th=[21376],
     | 99.99th=[27008]
    bw (KB/s)  : min=  528, max= 4296, per=3.55%, avg=2506.62, stdev=1353.48
    lat (usec) : 500=0.03%, 750=6.25%, 1000=18.47%
    lat (msec) : 2=54.42%, 4=15.98%, 10=4.51%, 20=0.28%, 50=0.06%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=0.16%, sys=0.23%, ctx=640409, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=640000/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=22415: Wed Sep 24 20:59:53 2014
  write: io=1978.4MB, bw=2278.2KB/s, iops=569 , runt=889245msec
    clat (usec): min=391 , max=99803 , avg=1754.59, stdev=1592.58
     lat (usec): min=391 , max=99803 , avg=1754.75, stdev=1592.57
    clat percentiles (usec):
     |  1.00th=[  708],  5.00th=[  756], 10.00th=[  812], 20.00th=[ 1020],
     | 30.00th=[ 1064], 40.00th=[ 1096], 50.00th=[ 1128], 60.00th=[ 1144],
     | 70.00th=[ 1336], 80.00th=[ 2736], 90.00th=[ 3280], 95.00th=[ 4384],
     | 99.00th=[ 7584], 99.50th=[ 9408], 99.90th=[17792], 99.95th=[21888],
     | 99.99th=[28800]
    bw (KB/s)  : min=  532, max= 4160, per=3.23%, avg=2281.68, stdev=1308.15
    lat (usec) : 500=0.01%, 750=3.50%, 1000=14.63%
    lat (msec) : 2=55.88%, 4=19.93%, 10=5.66%, 20=0.33%, 50=0.07%
    lat (msec) : 100=0.01%
  cpu          : usr=0.15%, sys=0.21%, ctx=508024, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=506464/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=22416: Wed Sep 24 20:59:53 2014
  write: io=2500.0MB, bw=2502.4KB/s, iops=625 , runt=1023031msec
    clat (usec): min=346 , max=92617 , avg=1597.26, stdev=1482.87
     lat (usec): min=346 , max=92618 , avg=1597.43, stdev=1482.87
    clat percentiles (usec):
     |  1.00th=[  668],  5.00th=[  740], 10.00th=[  764], 20.00th=[  956],
     | 30.00th=[ 1032], 40.00th=[ 1080], 50.00th=[ 1096], 60.00th=[ 1128],
     | 70.00th=[ 1176], 80.00th=[ 2320], 90.00th=[ 3120], 95.00th=[ 3952],
     | 99.00th=[ 7136], 99.50th=[ 8896], 99.90th=[17024], 99.95th=[21376],
     | 99.99th=[28288]
    bw (KB/s)  : min=  525, max= 4328, per=3.55%, avg=2506.39, stdev=1354.69
    lat (usec) : 500=0.03%, 750=6.26%, 1000=18.48%
    lat (msec) : 2=54.42%, 4=15.96%, 10=4.51%, 20=0.28%, 50=0.06%
    lat (msec) : 100=0.01%
  cpu          : usr=0.16%, sys=0.24%, ctx=640456, majf=0, minf=24
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=640000/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=22417: Wed Sep 24 20:59:53 2014
  write: io=1949.9MB, bw=2265.6KB/s, iops=566 , runt=881300msec
    clat (usec): min=391 , max=92616 , avg=1764.32, stdev=1598.58
     lat (usec): min=391 , max=92617 , avg=1764.49, stdev=1598.57
    clat percentiles (usec):
     |  1.00th=[  716],  5.00th=[  756], 10.00th=[  828], 20.00th=[ 1020],
     | 30.00th=[ 1064], 40.00th=[ 1096], 50.00th=[ 1128], 60.00th=[ 1144],
     | 70.00th=[ 1352], 80.00th=[ 2736], 90.00th=[ 3280], 95.00th=[ 4384],
     | 99.00th=[ 7584], 99.50th=[ 9408], 99.90th=[17792], 99.95th=[21888],
     | 99.99th=[29568]
    bw (KB/s)  : min=  508, max= 4064, per=3.22%, avg=2268.82, stdev=1304.84
    lat (usec) : 500=0.01%, 750=3.31%, 1000=14.45%
    lat (msec) : 2=55.91%, 4=20.23%, 10=5.69%, 20=0.34%, 50=0.07%
    lat (msec) : 100=0.01%
  cpu          : usr=0.15%, sys=0.21%, ctx=500079, majf=0, minf=24
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=499160/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=22418: Wed Sep 24 20:59:53 2014
  write: io=2500.0MB, bw=2502.2KB/s, iops=625 , runt=1023128msec
    clat (usec): min=364 , max=100200 , avg=1597.44, stdev=1485.80
     lat (usec): min=365 , max=100200 , avg=1597.60, stdev=1485.80
    clat percentiles (usec):
     |  1.00th=[  668],  5.00th=[  740], 10.00th=[  764], 20.00th=[  956],
     | 30.00th=[ 1032], 40.00th=[ 1080], 50.00th=[ 1096], 60.00th=[ 1128],
     | 70.00th=[ 1176], 80.00th=[ 2320], 90.00th=[ 3120], 95.00th=[ 3952],
     | 99.00th=[ 7136], 99.50th=[ 8896], 99.90th=[17024], 99.95th=[21376],
     | 99.99th=[27008]
    bw (KB/s)  : min=  545, max= 4320, per=3.55%, avg=2505.31, stdev=1355.22
    lat (usec) : 500=0.03%, 750=6.28%, 1000=18.48%
    lat (msec) : 2=54.42%, 4=15.91%, 10=4.54%, 20=0.28%, 50=0.06%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=0.16%, sys=0.24%, ctx=640965, majf=0, minf=24
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=640000/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=22419: Wed Sep 24 20:59:53 2014
  write: io=2500.0MB, bw=2502.3KB/s, iops=625 , runt=1023093msec
    clat (usec): min=354 , max=101809 , avg=1597.41, stdev=1486.66
     lat (usec): min=354 , max=101809 , avg=1597.57, stdev=1486.66
    clat percentiles (usec):
     |  1.00th=[  668],  5.00th=[  740], 10.00th=[  764], 20.00th=[  956],
     | 30.00th=[ 1032], 40.00th=[ 1080], 50.00th=[ 1096], 60.00th=[ 1128],
     | 70.00th=[ 1176], 80.00th=[ 2320], 90.00th=[ 3120], 95.00th=[ 3952],
     | 99.00th=[ 7136], 99.50th=[ 8896], 99.90th=[17024], 99.95th=[21376],
     | 99.99th=[27520]
    bw (KB/s)  : min=  526, max= 4296, per=3.55%, avg=2505.12, stdev=1355.00
    lat (usec) : 500=0.03%, 750=6.26%, 1000=18.47%
    lat (msec) : 2=54.43%, 4=15.93%, 10=4.53%, 20=0.28%, 50=0.06%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=0.16%, sys=0.23%, ctx=641309, majf=0, minf=24
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=640000/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=22420: Wed Sep 24 20:59:53 2014
  write: io=2500.0MB, bw=2502.4KB/s, iops=625 , runt=1023039msec
    clat (usec): min=359 , max=113082 , avg=1597.31, stdev=1484.54
     lat (usec): min=359 , max=113082 , avg=1597.47, stdev=1484.54
    clat percentiles (usec):
     |  1.00th=[  668],  5.00th=[  740], 10.00th=[  772], 20.00th=[  956],
     | 30.00th=[ 1032], 40.00th=[ 1080], 50.00th=[ 1096], 60.00th=[ 1128],
     | 70.00th=[ 1176], 80.00th=[ 2320], 90.00th=[ 3120], 95.00th=[ 3952],
     | 99.00th=[ 7136], 99.50th=[ 8896], 99.90th=[16768], 99.95th=[21376],
     | 99.99th=[27264]
    bw (KB/s)  : min=  546, max= 4304, per=3.55%, avg=2505.62, stdev=1352.68
    lat (usec) : 500=0.03%, 750=6.22%, 1000=18.38%
    lat (msec) : 2=54.53%, 4=16.00%, 10=4.50%, 20=0.28%, 50=0.06%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=0.16%, sys=0.23%, ctx=640486, majf=0, minf=24
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=640000/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=22421: Wed Sep 24 20:59:53 2014
  write: io=2500.0MB, bw=2500.5KB/s, iops=625 , runt=1023834msec
    clat (usec): min=360 , max=111553 , avg=1598.44, stdev=1482.34
     lat (usec): min=360 , max=111553 , avg=1598.68, stdev=1482.34
    clat percentiles (usec):
     |  1.00th=[  668],  5.00th=[  740], 10.00th=[  772], 20.00th=[  956],
     | 30.00th=[ 1032], 40.00th=[ 1080], 50.00th=[ 1096], 60.00th=[ 1128],
     | 70.00th=[ 1176], 80.00th=[ 2320], 90.00th=[ 3120], 95.00th=[ 3952],
     | 99.00th=[ 7136], 99.50th=[ 8896], 99.90th=[17024], 99.95th=[21120],
     | 99.99th=[27776]
    bw (KB/s)  : min=  531, max= 5240, per=3.55%, avg=2503.22, stdev=1352.29
    lat (usec) : 500=0.06%, 750=6.18%, 1000=18.29%
    lat (msec) : 2=54.66%, 4=15.93%, 10=4.53%, 20=0.27%, 50=0.06%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=0.17%, sys=0.23%, ctx=642125, majf=0, minf=24
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=640000/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=22422: Wed Sep 24 20:59:53 2014
  write: io=1996.2MB, bw=2285.3KB/s, iops=571 , runt=894444msec
    clat (usec): min=375 , max=108027 , avg=1749.14, stdev=1590.44
     lat (usec): min=375 , max=108027 , avg=1749.30, stdev=1590.43
    clat percentiles (usec):
     |  1.00th=[  708],  5.00th=[  756], 10.00th=[  812], 20.00th=[ 1020],
     | 30.00th=[ 1064], 40.00th=[ 1096], 50.00th=[ 1112], 60.00th=[ 1144],
     | 70.00th=[ 1336], 80.00th=[ 2736], 90.00th=[ 3280], 95.00th=[ 4384],
     | 99.00th=[ 7520], 99.50th=[ 9408], 99.90th=[18048], 99.95th=[21888],
     | 99.99th=[28544]
    bw (KB/s)  : min=  534, max= 4191, per=3.24%, avg=2288.39, stdev=1310.56
    lat (usec) : 500=0.01%, 750=3.56%, 1000=14.76%
    lat (msec) : 2=55.90%, 4=19.79%, 10=5.59%, 20=0.33%, 50=0.07%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=0.15%, sys=0.21%, ctx=512782, majf=0, minf=22
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=511004/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=22423: Wed Sep 24 20:59:53 2014
  write: io=1880.6MB, bw=2236.5KB/s, iops=559 , runt=861028msec
    clat (usec): min=376 , max=112284 , avg=1787.26, stdev=1617.83
     lat (usec): min=376 , max=112284 , avg=1787.42, stdev=1617.82
    clat percentiles (usec):
     |  1.00th=[  716],  5.00th=[  764], 10.00th=[  852], 20.00th=[ 1020],
     | 30.00th=[ 1064], 40.00th=[ 1096], 50.00th=[ 1128], 60.00th=[ 1144],
     | 70.00th=[ 1384], 80.00th=[ 2768], 90.00th=[ 3312], 95.00th=[ 4448],
     | 99.00th=[ 7648], 99.50th=[ 9536], 99.90th=[18048], 99.95th=[21888],
     | 99.99th=[28032]
    bw (KB/s)  : min=  552, max= 3856, per=3.17%, avg=2239.40, stdev=1296.46
    lat (usec) : 500=0.01%, 750=3.17%, 1000=14.03%
    lat (msec) : 2=55.47%, 4=21.07%, 10=5.85%, 20=0.34%, 50=0.07%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=0.14%, sys=0.21%, ctx=482055, majf=0, minf=22
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=481416/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=22424: Wed Sep 24 20:59:53 2014
  write: io=2500.0MB, bw=2502.3KB/s, iops=625 , runt=1023069msec
    clat (usec): min=369 , max=101809 , avg=1597.33, stdev=1482.85
     lat (usec): min=369 , max=101810 , avg=1597.49, stdev=1482.85
    clat percentiles (usec):
     |  1.00th=[  668],  5.00th=[  740], 10.00th=[  764], 20.00th=[  956],
     | 30.00th=[ 1032], 40.00th=[ 1080], 50.00th=[ 1096], 60.00th=[ 1128],
     | 70.00th=[ 1176], 80.00th=[ 2320], 90.00th=[ 3120], 95.00th=[ 3952],
     | 99.00th=[ 7136], 99.50th=[ 8896], 99.90th=[16768], 99.95th=[21376],
     | 99.99th=[28544]
    bw (KB/s)  : min=  542, max= 4280, per=3.55%, avg=2505.50, stdev=1353.97
    lat (usec) : 500=0.03%, 750=6.24%, 1000=18.46%
    lat (msec) : 2=54.45%, 4=15.98%, 10=4.50%, 20=0.29%, 50=0.06%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=0.16%, sys=0.24%, ctx=641944, majf=0, minf=22
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=640000/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=22425: Wed Sep 24 20:59:53 2014
  write: io=2500.0MB, bw=2501.1KB/s, iops=625 , runt=1023215msec
    clat (usec): min=356 , max=111887 , avg=1597.55, stdev=1488.88
     lat (usec): min=357 , max=111887 , avg=1597.71, stdev=1488.88
    clat percentiles (usec):
     |  1.00th=[  668],  5.00th=[  740], 10.00th=[  764], 20.00th=[  956],
     | 30.00th=[ 1032], 40.00th=[ 1080], 50.00th=[ 1096], 60.00th=[ 1128],
     | 70.00th=[ 1176], 80.00th=[ 2320], 90.00th=[ 3120], 95.00th=[ 3952],
     | 99.00th=[ 7136], 99.50th=[ 8896], 99.90th=[16768], 99.95th=[21120],
     | 99.99th=[27520]
    bw (KB/s)  : min=  560, max= 4312, per=3.55%, avg=2505.83, stdev=1354.20
    lat (usec) : 500=0.03%, 750=6.25%, 1000=18.42%
    lat (msec) : 2=54.50%, 4=15.96%, 10=4.51%, 20=0.28%, 50=0.06%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=0.16%, sys=0.23%, ctx=640547, majf=0, minf=22
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=640000/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=22426: Wed Sep 24 20:59:53 2014
  write: io=2500.0MB, bw=2502.4KB/s, iops=625 , runt=1023045msec
    clat (usec): min=354 , max=113449 , avg=1597.31, stdev=1486.23
     lat (usec): min=355 , max=113449 , avg=1597.47, stdev=1486.23
    clat percentiles (usec):
     |  1.00th=[  668],  5.00th=[  740], 10.00th=[  764], 20.00th=[  956],
     | 30.00th=[ 1032], 40.00th=[ 1080], 50.00th=[ 1096], 60.00th=[ 1128],
     | 70.00th=[ 1176], 80.00th=[ 2320], 90.00th=[ 3120], 95.00th=[ 3952],
     | 99.00th=[ 7200], 99.50th=[ 8896], 99.90th=[17024], 99.95th=[21120],
     | 99.99th=[27776]
    bw (KB/s)  : min=  541, max= 4296, per=3.55%, avg=2505.89, stdev=1353.95
    lat (usec) : 500=0.03%, 750=6.25%, 1000=18.48%
    lat (msec) : 2=54.41%, 4=15.98%, 10=4.51%, 20=0.28%, 50=0.06%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=0.16%, sys=0.23%, ctx=641765, majf=0, minf=22
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=640000/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=22427: Wed Sep 24 20:59:53 2014
  write: io=2500.0MB, bw=2502.2KB/s, iops=625 , runt=1023121msec
    clat (usec): min=348 , max=112290 , avg=1597.43, stdev=1485.04
     lat (usec): min=348 , max=112290 , avg=1597.59, stdev=1485.04
    clat percentiles (usec):
     |  1.00th=[  668],  5.00th=[  740], 10.00th=[  772], 20.00th=[  956],
     | 30.00th=[ 1032], 40.00th=[ 1080], 50.00th=[ 1096], 60.00th=[ 1128],
     | 70.00th=[ 1176], 80.00th=[ 2320], 90.00th=[ 3120], 95.00th=[ 3952],
     | 99.00th=[ 7136], 99.50th=[ 8896], 99.90th=[17024], 99.95th=[21120],
     | 99.99th=[27776]
    bw (KB/s)  : min=  541, max= 4288, per=3.55%, avg=2505.29, stdev=1354.05
    lat (usec) : 500=0.03%, 750=6.21%, 1000=18.45%
    lat (msec) : 2=54.49%, 4=15.95%, 10=4.53%, 20=0.28%, 50=0.06%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=0.16%, sys=0.23%, ctx=641459, majf=0, minf=22
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=640000/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=22428: Wed Sep 24 20:59:53 2014
  write: io=2500.0MB, bw=2502.7KB/s, iops=625 , runt=1022923msec
    clat (usec): min=360 , max=106116 , avg=1597.10, stdev=1484.12
     lat (usec): min=360 , max=106116 , avg=1597.26, stdev=1484.11
    clat percentiles (usec):
     |  1.00th=[  668],  5.00th=[  740], 10.00th=[  772], 20.00th=[  956],
     | 30.00th=[ 1032], 40.00th=[ 1080], 50.00th=[ 1096], 60.00th=[ 1128],
     | 70.00th=[ 1176], 80.00th=[ 2320], 90.00th=[ 3120], 95.00th=[ 3952],
     | 99.00th=[ 7136], 99.50th=[ 8896], 99.90th=[16768], 99.95th=[21376],
     | 99.99th=[28032]
    bw (KB/s)  : min=  534, max= 4312, per=3.55%, avg=2505.97, stdev=1352.45
    lat (usec) : 500=0.03%, 750=6.21%, 1000=18.44%
    lat (msec) : 2=54.46%, 4=16.01%, 10=4.50%, 20=0.28%, 50=0.06%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=0.16%, sys=0.23%, ctx=640363, majf=0, minf=22
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=640000/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=22429: Wed Sep 24 20:59:53 2014
  write: io=1941.1MB, bw=2262.5KB/s, iops=565 , runt=878914msec
    clat (usec): min=376 , max=99791 , avg=1766.78, stdev=1602.29
     lat (usec): min=376 , max=99791 , avg=1766.95, stdev=1602.28
    clat percentiles (usec):
     |  1.00th=[  716],  5.00th=[  756], 10.00th=[  828], 20.00th=[ 1020],
     | 30.00th=[ 1064], 40.00th=[ 1096], 50.00th=[ 1128], 60.00th=[ 1144],
     | 70.00th=[ 1352], 80.00th=[ 2736], 90.00th=[ 3280], 95.00th=[ 4384],
     | 99.00th=[ 7584], 99.50th=[ 9408], 99.90th=[17792], 99.95th=[21888],
     | 99.99th=[28800]
    bw (KB/s)  : min=  543, max= 3968, per=3.21%, avg=2265.82, stdev=1301.79
    lat (usec) : 500=0.01%, 750=3.26%, 1000=14.39%
    lat (msec) : 2=55.87%, 4=20.34%, 10=5.73%, 20=0.33%, 50=0.07%
    lat (msec) : 100=0.01%
  cpu          : usr=0.15%, sys=0.21%, ctx=498331, majf=0, minf=22
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=497128/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=22430: Wed Sep 24 20:59:53 2014
  write: io=2500.0MB, bw=2502.3KB/s, iops=625 , runt=1023098msec
    clat (usec): min=346 , max=79910 , avg=1597.38, stdev=1479.67
     lat (usec): min=346 , max=79911 , avg=1597.54, stdev=1479.67
    clat percentiles (usec):
     |  1.00th=[  668],  5.00th=[  740], 10.00th=[  772], 20.00th=[  956],
     | 30.00th=[ 1032], 40.00th=[ 1080], 50.00th=[ 1096], 60.00th=[ 1128],
     | 70.00th=[ 1176], 80.00th=[ 2320], 90.00th=[ 3120], 95.00th=[ 3952],
     | 99.00th=[ 7136], 99.50th=[ 8896], 99.90th=[17280], 99.95th=[21376],
     | 99.99th=[28288]
    bw (KB/s)  : min=  547, max= 4312, per=3.55%, avg=2505.18, stdev=1352.14
    lat (usec) : 500=0.03%, 750=6.18%, 1000=18.40%
    lat (msec) : 2=54.54%, 4=16.00%, 10=4.51%, 20=0.27%, 50=0.06%
    lat (msec) : 100=0.01%
  cpu          : usr=0.16%, sys=0.23%, ctx=640516, majf=0, minf=23
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=640000/d=0, short=r=0/w=0/d=0
randwrite4k30: (groupid=0, jobs=1): err= 0: pid=22431: Wed Sep 24 20:59:53 2014
  write: io=1961.1MB, bw=2272.2KB/s, iops=568 , runt=884181msec
    clat (usec): min=385 , max=101805 , avg=1759.26, stdev=1596.33
     lat (usec): min=385 , max=101805 , avg=1759.42, stdev=1596.33
    clat percentiles (usec):
     |  1.00th=[  708],  5.00th=[  756], 10.00th=[  812], 20.00th=[ 1020],
     | 30.00th=[ 1064], 40.00th=[ 1096], 50.00th=[ 1128], 60.00th=[ 1144],
     | 70.00th=[ 1352], 80.00th=[ 2736], 90.00th=[ 3280], 95.00th=[ 4384],
     | 99.00th=[ 7584], 99.50th=[ 9408], 99.90th=[18048], 99.95th=[21888],
     | 99.99th=[28288]
    bw (KB/s)  : min=  559, max= 4144, per=3.23%, avg=2276.10, stdev=1306.42
    lat (usec) : 500=0.01%, 750=3.44%, 1000=14.55%
    lat (msec) : 2=55.80%, 4=20.15%, 10=5.65%, 20=0.33%, 50=0.08%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=0.15%, sys=0.21%, ctx=502582, majf=0, minf=24
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=502244/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
  WRITE: io=70541MB, aggrb=70552KB/s, minb=2236KB/s, maxb=2502KB/s, mint=861028msec, maxt=1023834msec

sudo mkdir -p /backup/fio && sudo fio seqread.fio       && sudo rm -rf /backup/fio
seqread: (g=0): rw=read, bs=1M-1M/1M-1M/1M-1M, ioengine=sync, iodepth=1
fio-2.0.13
Starting 1 process
seqread: Laying out IO file(s) (1 file(s) / 10240MB)
Jobs: 1 (f=1): [R] [100.0% done] [88064K/0K/0K /s] [86 /0 /0  iops] [eta 00m:00s]
seqread: (groupid=0, jobs=1): err= 0: pid=22467: Wed Sep 24 21:11:08 2014
  read : io=51200MB, bw=90306KB/s, iops=88 , runt=580567msec
    clat (msec): min=8 , max=36 , avg=11.34, stdev= 1.44
     lat (msec): min=8 , max=36 , avg=11.34, stdev= 1.44
    clat percentiles (usec):
     |  1.00th=[ 9024],  5.00th=[ 9024], 10.00th=[ 9024], 20.00th=[ 9152],
     | 30.00th=[10944], 40.00th=[12096], 50.00th=[12096], 60.00th=[12224],
     | 70.00th=[12352], 80.00th=[12352], 90.00th=[12352], 95.00th=[12480],
     | 99.00th=[12480], 99.50th=[12480], 99.90th=[12864], 99.95th=[24192],
     | 99.99th=[31616]
    bw (KB/s)  : min=39309, max=113328, per=100.00%, avg=90388.94, stdev=11106.38
    lat (msec) : 10=26.38%, 20=73.57%, 50=0.06%
  cpu          : usr=0.03%, sys=0.42%, ctx=102125, majf=0, minf=282
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=51200/w=0/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
   READ: io=51200MB, aggrb=90306KB/s, minb=90306KB/s, maxb=90306KB/s, mint=580567msec, maxt=580567msec

sudo mkdir -p /backup/fio && sudo fio seqwrite.fio      && sudo rm -rf /backup/fio
seqwrite: (g=0): rw=write, bs=1M-1M/1M-1M/1M-1M, ioengine=sync, iodepth=1
fio-2.0.13
Starting 1 process
seqwrite: Laying out IO file(s) (1 file(s) / 10240MB)
Jobs: 1 (f=1): [W] [100.0% done] [0K/83968K/0K /s] [0 /82 /0  iops] [eta 00m:00s]
seqwrite: (groupid=0, jobs=1): err= 0: pid=22481: Wed Sep 24 21:21:58 2014
  write: io=51200MB, bw=81399KB/s, iops=79 , runt=644100msec
    clat (msec): min=11 , max=53 , avg=12.54, stdev= 1.30
     lat (msec): min=12 , max=53 , avg=12.58, stdev= 1.30
    clat percentiles (usec):
     |  1.00th=[12096],  5.00th=[12096], 10.00th=[12096], 20.00th=[12096],
     | 30.00th=[12224], 40.00th=[12224], 50.00th=[12224], 60.00th=[12224],
     | 70.00th=[12480], 80.00th=[12480], 90.00th=[12736], 95.00th=[14272],
     | 99.00th=[17536], 99.50th=[21888], 99.90th=[25984], 99.95th=[29568],
     | 99.99th=[37632]
    bw (KB/s)  : min=66650, max=84661, per=100.00%, avg=81475.29, stdev=2986.39
    lat (msec) : 20=99.41%, 50=0.59%, 100=0.01%
  cpu          : usr=0.40%, sys=0.37%, ctx=51247, majf=0, minf=23
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=51200/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
  WRITE: io=51200MB, aggrb=81398KB/s, minb=81398KB/s, maxb=81398KB/s, mint=644100msec, maxt=644100msec

sudo mkdir -p /backup/fio && sudo fio randread4k.fio    && sudo rm -rf /backup/fio
randread4k: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=1
fio-2.0.13
Starting 1 process
randread4k: Laying out IO file(s) (1 file(s) / 1024MB)
Jobs: 1 (f=1): [r] [100.0% done] [7464K/0K/0K /s] [1866 /0 /0  iops] [eta 00m:00s]
randread4k: (groupid=0, jobs=1): err= 0: pid=22497: Wed Sep 24 21:31:21 2014
  read : io=5120.0MB, bw=9495.6KB/s, iops=2373 , runt=552142msec
    clat (usec): min=276 , max=1251 , avg=420.01, stdev=50.77
     lat (usec): min=276 , max=1251 , avg=420.08, stdev=50.77
    clat percentiles (usec):
     |  1.00th=[  350],  5.00th=[  374], 10.00th=[  374], 20.00th=[  374],
     | 30.00th=[  422], 40.00th=[  422], 50.00th=[  422], 60.00th=[  422],
     | 70.00th=[  422], 80.00th=[  422], 90.00th=[  426], 95.00th=[  450],
     | 99.00th=[  620], 99.50th=[  628], 99.90th=[  748], 99.95th=[  748],
     | 99.99th=[  804]
    bw (KB/s)  : min= 6432, max= 9720, per=100.00%, avg=9499.85, stdev=702.99
    lat (usec) : 500=95.59%, 750=4.36%, 1000=0.04%
    lat (msec) : 2=0.01%
  cpu          : usr=0.47%, sys=0.78%, ctx=1594722, majf=0, minf=25
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=1310720/w=0/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
   READ: io=5120.0MB, aggrb=9495KB/s, minb=9495KB/s, maxb=9495KB/s, mint=552142msec, maxt=552142msec

sudo mkdir -p /backup/fio && sudo fio randwrite4k.fio   && sudo rm -rf /backup/fio
randwrite4k: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=1
fio-2.0.13
Starting 1 process
randwrite4k: Laying out IO file(s) (1 file(s) / 1024MB)
Jobs: 1 (f=1): [w] [100.0% done] [0K/5620K/0K /s] [0 /1405 /0  iops] [eta 00m:00s]
randwrite4k: (groupid=0, jobs=1): err= 0: pid=22511: Wed Sep 24 21:47:21 2014
  write: io=5120.0MB, bw=5469.1KB/s, iops=1367 , runt=958500msec
    clat (usec): min=348 , max=28708 , avg=729.53, stdev=195.07
     lat (usec): min=348 , max=28708 , avg=729.71, stdev=195.12
    clat percentiles (usec):
     |  1.00th=[  422],  5.00th=[  422], 10.00th=[  430], 20.00th=[  684],
     | 30.00th=[  708], 40.00th=[  724], 50.00th=[  740], 60.00th=[  748],
     | 70.00th=[  756], 80.00th=[  764], 90.00th=[ 1096], 95.00th=[ 1128],
     | 99.00th=[ 1144], 99.50th=[ 1160], 99.90th=[ 1176], 99.95th=[ 1208],
     | 99.99th=[ 1384]
    bw (KB/s)  : min= 4392, max= 6720, per=100.00%, avg=5473.98, stdev=564.10
    lat (usec) : 500=15.72%, 750=44.23%, 1000=29.42%
    lat (msec) : 2=10.63%, 4=0.01%, 10=0.01%, 50=0.01%
  cpu          : usr=0.52%, sys=0.91%, ctx=2231325, majf=0, minf=22
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=1310720/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
  WRITE: io=5120.0MB, aggrb=5469KB/s, minb=5469KB/s, maxb=5469KB/s, mint=958500msec, maxt=958500msec

sudo umount /backup


IOPS の結果だけまとめると以下のとおりです。File Storage (Consistent Performance) については、100 IOPS のものでテストしましたが、6000 IOPS のものでテストすれば 6000 IOPS という結果になるものと想定できます。他の結果の計算式を当てはめると 90 IOPS でしたが、30個の整数の足し算であり、結果が100程度だと誤差が大きすぎるので、ここでは100と記載します。計測中に表示される平均値はほとんどの時間で100と表示されていたのは前に述べたとおりです。


File Storage (Consistent Performance) - 20GB 100 IOPS
  READ:    100
  Write:   100

仮想サーバ(冗長化 NFS サーバ)
  READ:   7955
  Write:  2112

物理サーバ(冗長化 NFS サーバ)
  READ:  30574
  Write: 18272


ここまでの結果から私見をまとめると、優先順位の高い順で以下のようになります。
単体の NFS ボリュームが必要という前提です。当然、コスト対効果重視です。


・0.8TB~2.5TB の容量が必要で、32GB の SSD を発注してもおまけで 1TB 弱の SSD が
 支給されることに依存してもよい場合は、物理サーバをお勧めします(筆者一押し)。
・性能最優先の場合は、物理サーバをお勧めします。
・2.5TB~8TB の容量と、価格とのバランスが取れた性能がほしい場合は、
 仮想サーバをお勧めします。
・0~8TB の容量が必要で時間課金のものが必要な場合は、仮想サーバをお勧めします。
・8TB~130TB の容量が必要な場合は物理サーバをお勧めします(※)。
・性能を最低限に抑えることが許容でき、100GB までの容量が欲しい場合は、
 File Storage (Consistent Performance) をお勧めします。
・2.5TB~8TB の容量が必要で、予算が確保できる場合は、物理サーバをお勧めします。
・とにかく、何が何でも、SoftLayer が標準提供する NFS サーバが欲しい場合は、
 File Storage (Consistent Performance) をお勧めします。


(※) まだ検証記事として全くといっていいほど触れていませんが、複数の冗長化された iSCSI ターゲットを物理マシンで用意し、冗長化された NAS ヘッドで束ねて利用する、という構成も考えられます。必要に応じて少しずつ iSCSI ターゲットを追加していけるメリットも併せ持った構成です。たくさんの HDD を搭載可能な筐体は割高であり、スケールアップよりもスケールアウトです。特に、筆者一押しの物理サーバ構成でスケールアウトするのがよいと思います。

今回の計測で一番の収穫は、SoftLayer の仮想サーバがいかに性能面で優れているか、という点について、認識を改める機会となったことです。

0 件のコメント:

コメントを投稿