The Fastest Way to do Rsync

-

CLI

dd if=/dev/urandom of=test-file bs=1048576 count=100
dd if=/dev/urandom of=test-fileG bs=1048576 count=1000

for cipher in arcfour256 arcfour128 blowfish-cbc arcfour ; do
 echo "$cipher"
 for try in 1 2 ; do
  scp -c "$cipher" -P2211 me@192.168.0.201:/bak/tmp/test-fileG /dev/null
 done
done


rsync -aHhP --numeric-ids --stats -e "ssh -p 2211" me@192.168.0.201:/bak/tmp/_sync/ .
[18.08MB/s]

[ssh -T -c arcfour -o Compression=yes]=>[26.12MB/s]

rsync -aHhP --numeric-ids --stats --delete -e "ssh -T -c arcfour -o Compression=no -x -p2211" me@192.168.0.201:/bak/tmp/_sync/ .
[37.25MB/s]

[DsG] nc -l -p 7000 | lz4c -d| tar xv
[Server] tar cf - \[philosophy-raws\]\[Vocaloid\] | lz4c -c0 | nc 192.168.0.3 7000
[seems 80MB/s]

The fastest remote directory rsync over ssh archival I can muster (40MB/s over 1gb NICs)

This creates an archive that does the following:

rsync (Everyone seems to like -z, but it is much slower for me)
  • a: archive mode - rescursive, preserves owner, preserves permissions, preserves modification times, preserves group, copies symlinks as symlinks, preserves device files.
  • H: preserves hard-links
  • A: preserves ACLs
  • X: preserves extended attributes
  • x: don’t cross file-system boundaries
  • v: increase verbosity
  • –numeric-ds: don’t map uid/gid values by user/group name
  • –delete: delete extraneous files from dest dirs (differential clean-up during sync)
  • –progress: show progress during transfer
ssh
  • T: turn off pseudo-tty to decrease cpu load on destination.
  • c arcfour: use the weakest but fastest SSH encryption. Must specify “Ciphers arcfour” in sshd_config on destination.
  • o Compression=no: Turn off SSH compression.
  • x: turn off X forwarding if it is on by default.
Original
rsync -aHAXxv --numeric-ids --delete --progress -e "ssh -T -c arcfour -o Compression=no -x" user@<source>:<source_dir> <dest_dir>
Flip
rsync -aHAXxv --numeric-ids --delete --progress -e "ssh -T -c arcfour -o Compression=no -x" [source_dir] [dest_host:/dest_dir]

OpenSSH ciphers performance benchmark

In order to examine their performance, we will transfer the test file twice using each of the ciphers and note the transfer speed and delta. Here are the shell commands that we used:
for cipher in aes128-ctr aes192-ctr aes256-ctr arcfour256 arcfour128 aes128-cbc 3des-cbc blowfish-cbc cast128-cbc aes192-cbc aes256-cbc arcfour ; do
 echo "$cipher"
 for try in 1 2 ; do
  scp -c "$cipher" test-file root@192.168.100.102:
 done
done
You can review the raw results in the “ssh-cipher-speed-results.txt” file. The delta difference between the one and same benchmark test is within 16%-20%. Not perfect, but still enough for our tests.
Here is a chart which visualizes the results:
http://famzah.files.wordpress.com/2010/06/ssh-cipher-speed-chart.png

Comments

Post a Comment

Popular posts from this blog

SOX - Sound eXchange - How to use SOX for audio processing tasks in research.

-
This article is part of the on-going Software for Geeks series. SoX stands for So und e X change. SoX is a cross-platform command line audio utility tool that works on Linux, Windows and MacOS. It is very helpful in the following areas while dealing with audio and music files. Audio File Converter Editing audio files Changing audio attributes Adding audio effects Plus lot of advanced sound manipulation features In general, audio data is described by following four characteristics: Rate – The sample rate is in samples per second. For example, 44100/8000 Data size – The precision the data is stored in.  For example, 8/16 bits Data encoding – What encoding the data type uses. For example, u-law,a-law Channels – How many channels are contained in the audio data.  For example, Stereo 2 channels SoX supports over 20 audio file formats. To get the list of all supported formats, execute sox -h from the command line. One of the major benefits of a command-line audio/mu
Read more

Sox of Silence - Original post - http://digitalcardboard.com/blog/2009/08/25/the-sox-of-silence/

-
Image
The SoX of Silence SoX is, by their own definition,  the Swiss Army knife of audio manipulation. And no doubt it’s full of fun with slicing and dicing and playback and recording and filtering and effects capabilities. But SoX is a command line tool, which means obscure syntax and parameters in order to get things done. I’ve been trying off and on for months to try to understand the  silence  filter from within SoX, which allows one to remove silence from the beginning, middle, or end of the audio. Sounds, simple, doesn’t it?  Well, it should be. Below is the  man page for the  silence  filter : silence  [-l] above-periods [duration threshold[d|%] [below-periods duration threshold[d|%]] Removes silence from the beginning, middle, or end of the audio. Silence is anything below a specified threshold. The above-periods value is used to indicate if audio should be trimmed at the beginning of the audio. A value of zero indicates no silence should be trimmed from the beg
Read more

How to get video or audio duration of a file using ffmpeg?

-
Today we will learn How to get the duration of a file using ffmpeg 1. Format Container Duration - [This works for both Video and Audio Files - better to report numbers using this command] ffprobe -v error -show_entries format=duration \ -of default=noprint_wrappers=1:nokey=1 input.mp4     2. Video Stream Duration - [This works only for Videos]   ffprobe -v error -select_streams v:0 -show_entries stream=duration \ -of default=noprint_wrappers=1:nokey=1 input.mp4     Content author - https://superuser.com/questions/650291/how-to-get-video-duration-in-seconds
Read more