{"id":4675,"date":"2013-11-08T10:41:52","date_gmt":"2013-11-08T02:41:52","guid":{"rendered":"http:\/\/www.orczhou.com\/?p=4675"},"modified":"2013-11-10T13:42:44","modified_gmt":"2013-11-10T05:42:44","slug":"tranfer-data-faster-on-the-fly","status":"publish","type":"post","link":"https:\/\/www.orczhou.com\/index.php\/2013\/11\/tranfer-data-faster-on-the-fly\/","title":{"rendered":"\u4f7f\u7528tar+lz4\/pigz+ssh\u66f4\u5feb\u7684\u6570\u636e\u4f20\u8f93"},"content":{"rendered":"

\u524d\u9762\u4e00\u7bc7\u4ecb\u7ecd\u4e86\u5982\u4f55\u6700\u5927\u9650\u5ea6\u7684\u69a8\u53d6SCP\u7684\u4f20\u8f93\u901f\u5ea6<\/a>\uff0c\u6709\u4e86\u8fd9\u4e2a\u57fa\u7840\uff0c\u5c31\u53ef\u4ee5\u8fdb\u4e00\u6b65\u7684\u4f7f\u7528\u538b\u7f29\u6765\u52a0\u901f\u4f20\u8f93\u901f\u5ea6\u4e86\u3002\u53ea\u4f7f\u7528scp\uff0c\u4f20\u8f93\u901f\u7387\u6700\u5feb\u7ea690MB\uff0c\u672c\u6587\u901a\u8fc7\u538b\u7f29\u5c06\u628a\u6700\u5feb\u4f20\u8f93\u901f\u7387\u63d0\u5347\u5230\u7ea6250MB\/s(\u5305\u62ec\u89e3\u538b\u7684\u8fc7\u7a0b)\u3002<\/p>\n\n

1. \u7ed3\u8bba<\/h3>\n

\u4f7f\u7528tar+lz4+ssh\u7684\u65b9\u5f0f\u80fd\u591f\u83b7\u5f97\u6700\u5927\u7684\u4f20\u8f93\u6027\u80fd\uff1a<\/p>\n

time tar -c sendlog\/|pv|lz4 -B4|ssh -c arcfour128  \\\r\n-o\"MACs umac-64@openssh.com\" 10.xxx.xxx.36 \"lz4 -d |tar -xC \/u01\/backup_supu\"\r\n3.91GiB 0:00:16 [ 249MiB\/s] \r\n\r\nreal\t0m16.067s\r\nuser\t0m15.553s\r\nsys\t0m16.821s<\/div><\/pre>\n
249MB\/s\uff0c\u59a5\u59a5\u7684\u3002\u662f\u6700\u539f\u59cbscp(40MB\/s)\u76846\u500d\uff0c\u539f\u6765400GB\u4f20\u8f93\u9700\u8981\u7ea63\u5c0f\u65f6\uff0c\u73b0\u5728\u53ea\u9700\u898127\u5206\u949f\u4e86\u3002<\/strong><\/p>\n
\u6ce81\uff1alz4\u5728\u89e3\u538b\u65b9\u9762\u7684\u4f18\u5f02\u8868\u73b0\uff0c\u4f7f\u5f97\u4ed6\u5728\u672c\u6848\u4f8b\u4e2d\u975e\u5e38\u91cd\u8981\u3002\u5982\u679c\u65e0\u9700\u89e3\u538b\u7684\u4f20\u8f93\uff0c\u5219\u53ef\u4ee5\u8003\u8651\u4f7f\u7528pigz\/pbiz2<\/p>\n
\u6ce82\uff1a\u4f7f\u7528pv\u89c2\u5bdf\uff0c\u7f51\u7edc\u6d41\u91cf\u7ea680MB\uff0c\u6240\u4ee5\u4f7f\u7528nc\u66ff\u6362ssh\u5e76\u4e0d\u4f1a\u6709\u660e\u663e\u7684\u6027\u80fd\u63d0\u5347<\/p>\n
\u6ce83\uff1alz4\u538b\u7f29\u4f7f\u7528-B4(64KB\u5757\u5927\u5c0f)\uff0c\u89e3\u538b\u4f7f\u7528-B7(4MB\u5757\u5927\u5c0f)\uff0c\u662f\u672c\u6848\u4f8b\u7684\u6d4b\u8bd5\u6700\u4f18\u503c<\/p>\n
2. \u5173\u4e8elz4<\/h3>\n
lz4\u662f\u4e00\u4e2a\u8ba9”\u4eba\u89c1\u4eba\u7231\u3001\u82b1\u89c1\u82b1\u5f00”\u7684\u538b\u7f29\u7b97\u6cd5\uff0c\u80fd\u591f\u5728\u591a\u6838\u4e0a\u5f88\u597d\u7684\u6269\u5c55\uff0c\u538b\u7f29\u901f\u5ea6\u548c\u538b\u7f29\u6bd4\u5e76\u6ca1\u6709\u592a\u5927\u4f18\u52bf(pigz)\uff0c\u4f46\u662f\u4ed6\u7684\u89e3\u538b\u901f\u5ea6\u975e\u5e38\u60ca\u4eba\uff0c\u672c\u6848\u4f8b\u6d4b\u8bd5lz4\u7684\u89e3\u538b\u662fgunzip\u76843\u500d(\u66f4\u591a\u7684\u5bf9\u6bd4\u6d4b\u8bd5<\/a>)\u3002\u56e0\u4e3a\u538b\u7f29\u65f6\u9ad8\u6548\u7684\u591a\u6838\u5229\u7528\uff0c\u518d\u52a0\u4e0a\u60ca\u8273\u7684\u89e3\u538b\uff0clz4\u5df2\u7ecf\u5728\u975e\u5e38\u591a\u91cd\u8981\u573a\u5408\u4f7f\u7528\u4e86\uff1aLinux3.11\u5185\u6838\u5b9e\u73b0\u4e86LZ4\uff0c\u5e76\u53ef\u4ee5\u4f7f\u7528\u5176\u538b\u7f29\u548c\u89e3\u538bkernel image<\/a> HBase:Add an LZ4 compression option to HFile<\/a>\u7b49\u7b49(\u53c2\u8003<\/a>)\u3002<\/p>\n
\u5bf9\u4e8e\u9700\u8981\u9891\u7e41\u538b\u7f29\u3001\u5b9e\u65f6\u5feb\u901f\u89e3\u538b<\/strong>\u7684\u573a\u666f\u6765\u8bf4\uff0clz4\u975e\u5e38\u9002\u5408\u3002<\/p>\n
3. \u6027\u80fd\u73af\u5883\u8bf4\u660e<\/h3>\n
\u8fd9\u91cc\u4f7f\u7528\u540c\u4e0a\u4e00\u7bc7\u6587\u7ae0\u76f8\u540c\u7684\u4e24\u53f0\u4e3b\u673a\u73af\u5883\uff1aping\u83b7\u5f97RTT\u662f17ms\uff1b\u4f7f\u7528iperf\u6d4b\u8bd5\u5e26\u5bbd\u662f115MB(\u53c2\u8003\u9644\u5f55)\uff1b<\/p>\n
\u6574\u4e2a\u8fc7\u7a0b\u6709\u51e0\u4e2a\u9636\u6bb5\uff1a\u78c1\u76d8\u8bfb\u53d6–>\u6253\u5305(tar)–>\u538b\u7f29–>\u4f20\u8f93–>\u89e3\u538b\u7f29–>\u62c6\u5305–>\u843d\u76d8  \u5bf9\u5e94\u4e86\u7684\u901f\u5ea6\u6d4b\u8bd5\uff1a<\/p>\n
3.1 \u78c1\u76d8\u8bfb\u53d6\u548c\u843d\u76d8<\/h4>\n
\u78c1\u76d8\u8bfb\u53d6(\u6709page cache)\uff0c\u80fd\u52303GB\/s\uff1b\u78c1\u76d8\u5199\u5165\u7ea6428MB\uff1a<\/p>\n
# dd if=.\/sendlog.tar of=\/dev\/null bs=4096 count=1048576\r\n1024002+1 records in\r\n1024002+1 records out\r\n4194314240 bytes (4.2 GB) copied, 1.33946 s, 3.1 GB\/s\r\n\r\n# dd if=\/dev\/zero of=.\/x.zero.file bs=4096 count=1048576\r\n1048576+0 records in\r\n1048576+0 records out\r\n4294967296 bytes (4.3 GB) copied, 10.0306 s, 428 MB\/s<\/div><\/pre>\n
3.2 \u6253\u5305\u3001\u62c6\u5305<\/h4>\n
\u6253\u5305\u548c\u62c6\u5305\u901f\u5ea6\u90fd\u5927\u4e8e350MB\/s\uff1a<\/p>\n
# time tar -cf sendlog.tar .\/sendlog\/\r\nreal\t0m10.996s\r\n# time tar -xf sendlog.tar\r\nreal\t0m11.564s<\/div><\/pre>\n
3.3 \u538b\u7f29\u3001\u89e3\u538b\u7f29<\/h4>\n
\u5173\u4e8e\u5404\u4e2a\u538b\u7f29\u5de5\u5177\u7684\u6027\u80fd(\u538b\u7f29\u3001\u89e3\u538b\u3001\u538b\u7f29\u7387)\u5df2\u7ecf\u6709\u5f88\u591a\u4eba\u505a\u4e86\u6bd4\u8f83\uff0c\u672c\u6587\u4e0d\u505a\u8be6\u7ec6\u8ba8\u8bba\uff0c\u8fd9\u91cc\u9009\u62e9gzip\/pigz lz4 bzip\u505a\u672c\u6d4b\u8bd5\u7684\u6bd4\u8f83\uff1a<\/p>\n
           | input speed | output speed | rate   | speed of decoder\r\npigz -p 16 | 327.0MB\/s   | 57.2MB\/s     | 17.5%  | 95  MB\/s\r\nlz4        | 288.0MB\/s   | 79.2MB\/s     | 27.5%  | 264 MB\/s\r\nbzip2      |   4.9MB\/s   | 0.65MB\/s     | 13.1%  | 25.6MB \/s
<\/pre>\n
\u538b\u7f29\u5de5\u5177\u7684\u6bd4\u8f83\u6d4b\u8bd5\u53c2\u8003\uff1aGzip vs Bzip2 vs LZMA vs XZ vs LZ4 vs LZO<\/a><\/p>\n
\u53ef\u4ee5\u770b\u5230\uff0clz4\u5728\u538b\u7f29\u7387\u4e0a\u7565\u5fae\u900a\u8272(\u5bf9\u6bd4pigz)\uff0c\u4f46\u662f\u5728\u89e3\u538b\u901f\u5ea6\u4e0a\u6709\u8fd9\u60ca\u4eba<\/strong>\u7684\u4f18\u52bf\u3002<\/p>\n
3.4 \u4f20\u8f93<\/h4>\n
\u524d\u6587\u4ecb\u7ecd\u4e86scp<\/a>\uff0c\u7ea690MB\u6700\u5feb\u7684\u4f20\u8f93\u901f\u5ea6\u3002<\/p>\n
3.5 \u6574\u4f53\u6d41\u7a0b<\/h4>\n
\u78c1\u76d8\u8bfb\u53d6---->\u6253\u5305---->\u538b\u7f29------>\u4f20\u8f93---->\u89e3\u538b\u7f29-->\u62c6\u5305---->\u843d\u76d8\r\n             |->tar   |->gzip    |->ssh   |->gzip  |->tar\r\n                      |->bzip2   |->http  |->bzip\r\n                      |-> ...    |->nc    |->...\r\n                      |->lz4              |->lz4\r\n>400MB\/s    >350MB\/s  79MB\/s     90MB\/s   72MB\/s    >350MB\/s >400MB\/s<\/blockquote><\/pre>\n
\u8fd9\u91cc\u53ef\u4ee5\u770b\u5230\uff0c\u89e3\u538b\u662f\u6700\u5927\u7684\u74f6\u9888\uff0c\u4f7f\u7528\u5728\u89e3\u538b\u65b9\u9762\u6700\u6709\u4f18\u52bf\u7684\u538b\u7f29\u5de5\u5177\uff0c\u80fd\u8ba9\u4f20\u8f93\u83b7\u5f97\u6700\u5927\u901f\u5ea6\u3002<\/b>\u800clz4\u6b63\u662f\u5728\u89e3\u538b\u6548\u7387\u65b9\u9762\u6709\u7740\u5de8\u5927\u7684\u4f18\u52bf\u3002<\/p>\n
\u6309\u7167\u4e0a\u9762lz4\u7684\u6d4b\u8bd5\uff0c\u4f20\u8f93\u901f\u5ea6\u7406\u8bba\u503c\u4e3a264MB\/s(\u6b64\u65f6\u4f20\u8f93\u901f\u5ea6\u4e3a264*27.3%=72MB)\uff0c\u8fd9\u4e5f\u662f\u672c\u6b21\u6d4b\u8bd5\u7684\u7406\u8bba\u4e0a\u9650\u901f\u5ea6\u3002<\/p>\n
4. \u5b9e\u9a8c\u6d4b\u8bd5<\/h3>\n
\u4f7f\u7528lz4\u538b\u7f29\u4f20\u8f93\uff1a<\/p>\n
# time tar -c sendlog\/|lz4|ssh -c arcfour128 \\\r\n -o\"MACs umac-64@openssh.com\" 10.xxx.xx.36 \"lz4 -d |tar -xC \/u01\/backup_supu\"\r\nreal\t0m25.646s\r\nreal\t0m25.911s\r\nreal\t0m29.019s<\/div><\/pre>\n
\u6d4b\u8bd5\u4e09\u6b21\uff0c\u5206\u522b\u8017\u65f626s\u300129s\u300125.6s\uff0c\u4f20\u8f93\u7684\u5e73\u5747\u901f\u5ea6\u4e3a\uff1a152MB\/s\uff0c\u7f51\u7edc\u5e26\u5bbd\u5360\u7528\u7ea641.9MB\/s\u3002<\/p>\n
\u4f7f\u7528pigz\u7684\u538b\u7f29\u4f20\u8f93\uff1a<\/p>\n
# time tar -c sendlog\/|pigz -p 16|ssh -c arcfour128 \\\r\n -o\"MACs umac-64@openssh.com\" 10.xxx.xx.36 \"gzip -d|tar -xC \/u01\/backup_supu\"\r\nrreal\t0m37.030s\r\nreal\t0m25.911s\r\nreal\t0m29.019s<\/div><\/pre>\n
\u6d4b\u8bd5\u4e09\u6b21\uff0c\u5206\u522b\u8017\u65f637s\u300137.2s\u300135.6s\uff0c\u4f20\u8f93\u7684\u5e73\u5747\u901f\u5ea6\u4e3a\uff1a110.7MB\/s\uff0c\u7f51\u7edc\u5e26\u5bbd\u5360\u7528\u7ea619.4MB\/s\u3002<\/p>\n
\u5bf9\u6bd4\u53d1\u73b0\uff0c\u5728\u538b\u7f29\u65b9\u9762pigz\u4e0elz4\u5e76\u6ca1\u6709\u592a\u5927\u533a\u522b\uff0c\u4f46\u662flz4\u89e3\u538b\u901f\u5ea6\u975e\u5e38\u5feb\uff0c\u6240\u4ee5\u5728\u8fd9\u79cd\u9700\u8981\u7acb\u523b\u89e3\u538b\u7684\u573a\u666f\u4e0b\uff0clz4\u8f7b\u677e\u80dc\u51fa(bzip2\u8fd9\u79cd\u5c31\u4e0d\u9700\u8981\u6d4b\u8bd5\u4e86)\u3002<\/p>\n
4.1 \u5206\u6790<\/h4>\n
\u6309\u7167\u7b2c\u4e8c\u8282\u4e2d\u7684\u7406\u8bba\u5206\u6790\uff0c\u4f20\u8f93\u901f\u5ea6\u5e94\u8be5\u80fd\u5230260MB\uff0c\u4f46\u662f\u4e0a\u9762\u53ea\u6709152MB\/s\uff0c\u8fd9\u8bf4\u660e\uff0c\u8fd8\u6709\u8c03\u4f18\u7684\u7a7a\u95f4\u3002\u7ee7\u7eed\u5206\u6790\uff0c\u770b\u770b\u74f6\u9888\u5728\u54ea\u513f\uff1a<\/p>\n
\u4f7f\u7528pv\u5de5\u5177\u89c2\u5bdf\u5230\uff0ctar+lz4\u6709\u7ea670MB\/s\u7684\u8f93\u51fa\uff1a<\/p>\n
 time tar -c sendlog\/|lz4|pv > \/dev\/null\r\n1.02GiB 0:00:14 [70.8MiB\/s] [                                 <=>]<\/div><\/pre>\n
\u6bd4\u76f4\u63a5lz4\u8f93\u51fa\uff0c\u8981\u6162\u4e8610%\u5de6\u53f3(lz\u7ea679MB\/s)\u3002<\/p>\n
\u518d\u52a0\u4e0a\u4e00\u6b21\u7f51\u7edcssh\uff1a<\/p>\n
time tar -c sendlog\/|lz4|pv|ssh -c arcfour128 -o \"MACs umac-64@openssh.com\" 10.xxx.xxx.36 \"cat - >\/dev\/null\"\r\n1.02GiB 0:00:23 [43.9MiB\/s] [                                 <=>]<\/div><\/pre>\n
\u6bd4\u76f4\u63a5lz4\u8f93\u51fa\uff0c\u8981\u6162\u4e8645%\u5de6\u53f3(lz\u7ea679MB\/s)\uff1b\u8fdc\u7aef\u518d\u52a0\u4e0a\u89e3\u538b\u548c\u62c6\u5305\uff0c\u538b\u7f29\u540e\u7684\u4f20\u8f93\u901f\u5ea6\u5c31\u662f41.9MB\/s\u3002\u4e3a\u4ec0\u4e48\u4f1a\u4e0b\u964d\uff0c\u8fd8\u4e0d\u660e\u4e86\uff0c\u4f5c\u8005\u4e5f\u8fd8\u6ca1\u6709\u60f3\u5230\u6709\u4ec0\u4e48\u65b9\u6cd5\u80fd\u591f\u76f4\u63a5\u52a0\u901f\u8fd9\u6837\u7684\u7ba1\u9053\u4f20\u8f93\uff0c\u5982\u679c\u770b\u5ba2\u6709\u4ec0\u4e48\u5efa\u8bae\uff0c\u4e0d\u59a8\u5206\u4eab\uff0c\u770b\u770b\u8fd8\u80fd\u4e0d\u80fd\u4f18\u5316\uff0c\u7ee7\u7eed\u63d0\u5347\u901f\u5ea6\u3002<\/p>\n
\u81f3\u6b64\uff0c\u4f20\u8f93\u901f\u5ea6\u5c31\u80fd\u591f\u5230150MB\/s\u3002\u6bd4\u6700\u539f\u59cbscp(40MB\/s)\u8981\u5feb\u4e86\u7ea64\u500d\uff0c\u539f\u6765400GB\u9700\u8981\u7ea63\u5c0f\u65f6\uff0c\u73b0\u5728\u53ea\u9700\u898145\u5206\u949f\u4e86\u3002<\/strong><\/p>\n
5. lz4\u53c2\u6570\u6d4b\u8bd5<\/h3>\n
\u524d\u9762\u8bd5\u9a8c\u53d1\u73b0\uff0c\u6574\u4e2a\u6d41\u7a0b\u4e2dlz4\u538b\u7f29\u6bd4\u9884\u671f\u7684\u8981\u616245%\u5de6\u53f3\uff0c\u800c\u8fd9\u91cc\u533a\u522b\u4ec5\u4ec5\u662f\u4e00\u4e2a\u4f7f\u7528\u7ba1\u9053(pipe)\u3001\u4e00\u4e2a\u76f4\u63a5\u8bfb\u53d6\u3002\u8fd9\u91cc\u5c1d\u8bd5\u901a\u8fc7\u4fee\u6539lz4\u5757\u5927\u5c0f\u5bf9\u6bd4\uff0c\u662f\u5426\u6709\u6027\u80fd\u63d0\u5347\uff1a<\/p>\n
\"lz4-with-different-block-size\"<\/a><\/p>\n
\u6d4b\u8bd5\u547d\u4ee4\uff1a<\/p>\n
for i in `seq 4 7`; do time tar -c .\/sendlog\/|lz4 -B$i |pv > \/dev\/null ;done\r\n1.07GiB 0:00:11 [94.4MiB\/s] [                          <=>]\r\nreal\t0m11.640s\r\nuser\t0m10.375s\r\nsys\t0m4.308s<\/div><\/pre>\n
\u53ef\u4ee5\u770b\u5230\u5757\u5927\u5c0f\u4e3a64KB\u7684\u65f6\u5019\uff0clz\u7684\u538b\u7f29\u901f\u5ea6\u6709\u663e\u8457\u63d0\u5347(31%)\u3002\u4e8e\u662f\uff0c\u6211\u4eec\u5728lz4\u65b0\u589e\u53c2\u6570-B4\uff0c\u770b\u770b\u662f\u5426\u80fd\u591f\u63d0\u5347\u6027\u80fd\uff1a<\/p>\n
Bang!\u786e\u5b9e\uff0c\u4f20\u8f93\u6027\u80fd\u63d0\u5347\u5230\u4e86\u7ea6249MB\/s:<\/p>\n
time tar -c sendlog\/|pv|lz4 -B4|ssh -c arcfour128  \\\r\n-o\"MACs umac-64@openssh.com\" 10.xxx.xxx.36 \"lz4 -d |tar -xC \/u01\/backup_supu\"\r\n3.91GiB 0:00:16 [ 249MiB\/s] \r\n\r\nreal\t0m16.067s\r\nuser\t0m15.553s\r\nsys\t0m16.821s<\/div><\/pre>\n
5. \u4e3a\u4ec0\u4e48\u4e0d\u7528nc<\/h3>\n
\u5c31\u4e0d\u7528\u5b83!!!<\/p>\n
* nc\u4e0d\u6bd4ssh\u5feb\uff1b\u5982\u679c\u538b\u7f29\u540e\u4f20\u8f93\uff0cnc\u6bd4ssh\u6ca1\u6709\u4f18\u52bf<\/p>\n
* nc\u5728\u811a\u672c\u4e2d\u4e0d\u597d\u8c03\u7528\uff0c\u9700\u8981\u5728\u4e24\u7aef\u6267\u884c\u547d\u4ee4<\/p>\n
* nc\u9700\u8981\u4e00\u4e2a\u989d\u5916\u7684\u7f51\u7edc\u7aef\u53e3<\/p>\n
* nc\u4e0d\u52a0\u5bc6<\/p>\n
6. \u8fd8\u80fd\u4e0d\u80fd\u66f4\u5feb<\/h3>\n
\u672c\u6848\u4f8b\u4e2d\uff0clz4\u89e3\u538b\u7f29\u7684\u901f\u5ea6\u662f264MB\/s\uff0c\u8fd9\u91cc\u80fd\u591f\u8fbe\u5230249MB\/s\uff0c\u5e94\u8be5\u8fd8\u6709\u4e00\u70b9\u70b9\u53ef\u4ee5\u69a8\u53d6\uff0c\u4e0d\u8fc7\u6211\u5df2\u7ecf\u6ca1\u6709\u62db\u4e86\u3002<\/p>\n
\u9644\u5f55<\/h3>\n
iperf\u7684\u5e26\u5bbd\u6d4b\u8bd5\uff1a<\/p>\n
iperf -c 10.xxx.xx.18 -p 3999 -t 30\r\n------------------------------------------------------------\r\nClient connecting to 10.xxx.xx.18, TCP port 3999\r\nTCP window size: 16.0 KByte (default)\r\n------------------------------------------------------------\r\n[  3] local 10.xx.xx.36 port 43838 connected with 10.xx.xx.18 port 3999\r\n[ ID] Interval       Transfer     Bandwidth\r\n[  3]  0.0-30.0 sec  3.15 GBytes   903 Mbits\/sec\r\n\r\niperf -s -p 3999 -m\r\n------------------------------------------------------------\r\nServer listening on TCP port 3999\r\nTCP window size: 85.3 KByte (default)\r\n------------------------------------------------------------\r\n[  4] local 10.xx.xx.18 port 3999 connected with 10.xx.xx.36 port 43838\r\n[ ID] Interval       Transfer     Bandwidth\r\n[  4]  0.0-30.0 sec  3.15 GBytes   902 Mbits\/sec\r\n[  4] MSS size 1448 bytes (MTU 1500 bytes, ethernet)<\/div><\/pre>\n
\u53c2\u8003\u9605\u8bfb<\/h3>\n
* lz4@Google code<\/a><\/p>\n
* lz4’s details<\/a><\/p>\n
* LZ4 Streaming Format<\/a><\/p>\n
* Quick Benchmark: Gzip vs Bzip2 vs LZMA vs XZ vs LZ4 vs LZO<\/a><\/p>\n
* lz4: Extremely Fast Compression algorithm<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
\u524d\u9762\u4e00\u7bc7\u4ecb\u7ecd\u4e86\u5982\u4f55\u6700\u5927\u9650\u5ea6\u7684\u69a8\u53d6SCP\u7684\u4f20\u8f93\u901f\u5ea6\uff0c\u6709\u4e86\u8fd9\u4e2a\u57fa\u7840\uff0c\u5c31\u53ef\u4ee5\u8fdb\u4e00\u6b65\u7684\u4f7f\u7528\u538b\u7f29\u6765\u52a0\u901f\u4f20\u8f93\u901f\u5ea6\u4e86\u3002\u53ea\u4f7f\u7528scp\uff0c\u4f20\u8f93\u901f\u7387\u6700\u5feb\u7ea690MB\uff0c\u672c\u6587\u901a\u8fc7\u538b\u7f29\u5c06\u628a\u6700\u5feb\u4f20\u8f93\u901f\u7387\u63d0\u5347\u5230\u7ea6250MB\/s(\u5305\u62ec\u89e3\u538b\u7684\u8fc7\u7a0b)\u3002 1. \u7ed3\u8bba \u4f7f\u7528tar+lz4+ssh\u7684\u65b9\u5f0f\u80fd\u591f\u83b7\u5f97\u6700\u5927\u7684\u4f20\u8f93\u6027\u80fd\uff1a time tar -c sendlog\/|pv|lz4 -B4|ssh -c arcfour128 \\ -o”MACs umac-64@openssh.com” 10.xxx.xxx.36 “lz4 -d |tar -xC \/u01\/backup_supu” 3.91GiB 0:00:16 [ 249MiB\/s] real 0m16.067s user 0m15.553s sys 0m16.821s 249MB\/s\uff0c\u59a5\u59a5\u7684\u3002\u662f\u6700\u539f\u59cbscp(40MB\/s)\u76846\u500d\uff0c\u539f\u6765400GB\u4f20\u8f93\u9700\u8981\u7ea63\u5c0f\u65f6\uff0c\u73b0\u5728\u53ea\u9700\u898127\u5206\u949f\u4e86\u3002 \u6ce81\uff1alz4\u5728\u89e3\u538b\u65b9\u9762\u7684\u4f18\u5f02\u8868\u73b0\uff0c\u4f7f\u5f97\u4ed6\u5728\u672c\u6848\u4f8b\u4e2d\u975e\u5e38\u91cd\u8981\u3002\u5982\u679c\u65e0\u9700\u89e3\u538b\u7684\u4f20\u8f93\uff0c\u5219\u53ef\u4ee5\u8003\u8651\u4f7f\u7528pigz\/pbiz2 \u6ce82\uff1a\u4f7f\u7528pv\u89c2\u5bdf\uff0c\u7f51\u7edc\u6d41\u91cf\u7ea680MB\uff0c\u6240\u4ee5\u4f7f\u7528nc\u66ff\u6362ssh\u5e76\u4e0d\u4f1a\u6709\u660e\u663e\u7684\u6027\u80fd\u63d0\u5347 \u6ce83\uff1alz4\u538b\u7f29\u4f7f\u7528-B4(64KB\u5757\u5927\u5c0f)\uff0c\u89e3\u538b\u4f7f\u7528-B7(4MB\u5757\u5927\u5c0f)\uff0c\u662f\u672c\u6848\u4f8b\u7684\u6d4b\u8bd5\u6700\u4f18\u503c<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_eb_attr":"","inline_featured_image":false,"_tocer_settings":[],"footnotes":""},"categories":[27,6],"tags":[],"class_list":["post-4675","post","type-post","status-publish","format-standard","hentry","category-linux","category-mysql"],"_links":{"self":[{"href":"https:\/\/www.orczhou.com\/index.php\/wp-json\/wp\/v2\/posts\/4675","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.orczhou.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.orczhou.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.orczhou.com\/index.php\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.orczhou.com\/index.php\/wp-json\/wp\/v2\/comments?post=4675"}],"version-history":[{"count":17,"href":"https:\/\/www.orczhou.com\/index.php\/wp-json\/wp\/v2\/posts\/4675\/revisions"}],"predecessor-version":[{"id":4695,"href":"https:\/\/www.orczhou.com\/index.php\/wp-json\/wp\/v2\/posts\/4675\/revisions\/4695"}],"wp:attachment":[{"href":"https:\/\/www.orczhou.com\/index.php\/wp-json\/wp\/v2\/media?parent=4675"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.orczhou.com\/index.php\/wp-json\/wp\/v2\/categories?post=4675"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.orczhou.com\/index.php\/wp-json\/wp\/v2\/tags?post=4675"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}