nixCraft Linux Sys Admin Blog

OpenSSH ( SSHD ) Speed Optimization For Long Distance Data Transfer

Posted: 14 May 2008 06:02 PM CDT

One of my client has server node located at north America, Asia and Europe data centers. All servers are connected using 1000Mbps links. They transfers lots of data between all nodes over ssh session using scp / sftp. However, performance was horrible. After some research I came across High Performance SSH/SCP - HPN-SSH patch for OpenSSH:

SCP and the underlying SSH2 protocol implementation in OpenSSH is network performance limited by statically defined internal flow control buffers. These buffers often end up acting as a bottleneck for network throughput of SCP, especially on long and high bandwith network links.

Modifying the ssh code to allow the buffers to be defined at run time eliminates this bottleneck. We have created a patch that will remove the bottlenecks in OpenSSH and is fully interoperable with other servers and clients. In addition HPN clients will be able to download faster from non HPN servers, and HPN servers will be able to receive uploads faster from non HPN clients. However, the host receiving the data must have a properly tuned TCP/IP stack.

The amount of improvement any specific user will see is dependent on a number of issues. Transfer rates cannot exceed the capacity of the network nor the throughput of the I/O subsystem including the disk and memory speed. The improvement will also be highly influenced by the capacity of the processor to perform the encryption and decryption. Less computational expensive ciphers will often provide better throughput than more complex ciphers.

You can download HPN-SSH patch here. This patch improved our performance. You also need to tweak Linux TCP/IP networking settings. Here is my sysctl.conf file ( read this TCP tunning Linux guide for detailed explanation) :
net.ipv4.netfilter.ip_conntrack_max=103728 # optimization start # increase TCP max buffer size setable using setsockopt() net.ipv4.tcp_rmem = 4096 87380 8388608 net.ipv4.tcp_wmem = 4096 87380 8388608 # increase Linux auto tuning TCP buffer limits # min, default, and max number of bytes to use # set max to at least 4MB, or higher if you use very high BDP paths net.core.rmem_max = 8388608 net.core.wmem_max = 8388608 net.core.netdev_max_backlog = 5000 net.ipv4.tcp_window_scaling = 1 # optimization end

nixCraft Known Issues: Email Subscriptions and Print Page

Posted: 14 May 2008 05:37 PM CDT

Our email delivery system is powered by feedburner and they had some problem as a result there were no email update sent in last 2-3 days. However, feedburner restarted the polling process; starting now, any feed changes not yet delivered via email should be detected and delivered, albeit up to three days late, and from this point forward on your regularly scheduled window.

Also, thanks to readers' feedback print functionality on our site is restored. You can now get print view instantly. This was due to incomparability wordpress plugin issue. We regret this error!

Don't hesitate to contact me if you have any more problems.

Linux Success Story: Go Mumbai Prepaid Smart Card for BEST

Posted: 14 May 2008 05:26 PM CDT

Thanks to low cost and high reliability of Linux - many commuters have opted a smart and cashless way to pay fair every day. Mumbai is commercial and entertainment center of India. Public buses run by BEST cover almost all parts of the metropolis, as well as parts of Navi Mumbai and Thane. Buses are used for commuting short to medium distances, while train fares are more economical for long distance commutes. The BEST runs a total of 3,500 buses, ferrying 4.5 million passengers over 340 routes.

In the swarming Indian metropolis Mumbai, it can be a gymnastic exercise just to fish in your pocket on the packed city buses and stretch out your paying hand to the conductor. Many commuters have opted instead for a 'smart' and cashless way to pay, provided, in part, by Linux. Named Go Mumbai, it is a prepaid smart card for BEST (Brihanmumbai Electric Supply and Transport) bus journeys.

=> Linux rides pillion on Mumbai city buses

Download of the day: Fedora 9 CD / DVD ISO

Posted: 13 May 2008 05:00 PM CDT

Fedora Linux version 9 has been released and available for download ( jump to download link ). The Fedora Project is a Red Hat sponsored by Red Hat and guided by community and contributing members.

New in Fedora 9

This release includes many new features and software such as GNOME 2.22, KDE 4.0.3, Firefox 3 beta 5, PackageKit ( cross-distribution package management solution ), OpenJDK (Open Source Java version), SELinux confined web browser, Upstart init daemon, Xen virtualization, and many more cutting edge features.

(Fig.01 Fedora Linux 9 Desktop - Click to enlarge image)

Download link ( Download Fedora )

You can download Fedora via the web/ftp server or via BitTorrent (recommended).

Fedora 9 DVD ISO download

Visit mirror to download Fedora ISO images (32 bit DVD ISO ~ 3.3 GB)
Visit mirror to download Fedora ISO images (64 bit DVD ISO ~ 3.3 GB)

Fedora 9 Live CD ISO download

Use 32 bit Live CD (691M), if internet connection is slow or you just want to see how Fedora works. You can install fedora from Live cd.

Fedora 9 DVD ISO BitTorrent download

Download from BitTorrent i386 32 bit DVD (requires a BitTorrent client)
Download from BitTorrent 64 bit DVD (requires a BitTorrent client)

Security Warning: Serious flaw in Debian Linux OpenSSL Package

Posted: 13 May 2008 02:48 PM CDT

There is a serious security flaw in Debian openssl - the random number generator in Debian's openssl package is predictable. As a result, cryptographic key material may be guessable.

=> Package : openssl
=> Vulnerability : predictable random number generator
=> Problem type : remote
=> Debian-specific: yes
=> CVE Id(s) : CVE-2008-0166
=> Checkout description and recommended fix at the following url:

[SECURITY] [DSA 1571-1] New openssl packages fix predictable random number generator

CentOS 4 / Redhat Enterprise Linux 4 iSCSI Installation and Configuration ( iscsi initiators )

Posted: 13 May 2008 11:24 AM CDT

Next in series » »

« « Previous in series

I've received a couple of email about setting up iSCSI under CentOS 4 or RHEL ES 4 server. Previously, I wrote about iSCSI under CentOS 5 / RHEL 5 server.

Requirements

[a] Following instructions tested on RHEL ES 4 and CentOS 4 only. (See RHEL 5/ CentOS 5 / Debian/ Ubuntu Linux specific instructions here).
[b] You need following information
[c] ISCSI Username
[d] ISCSI Password
[e] ISCSI Server IP / hostname

CentOS Linux v4.x - Install iscsi-initiator-utils

Type the following command:
# yum install iscsi-initiator-utils

Redhat ES Linux v4.x - Install iscsi-initiator-utils

Type the following command:
# up2date iscsi-initiator-utils

Configure iSCSI

Open /etc/iscsi.conf file, enter:
# vi /etc/iscsi.conf
Setup it as follows:

DiscoveryAddress=ISCSI_TARGET_HOST_OR_IP  OutgoingUserName=ISCSI_USER_NAME  OutgoingPassword=ISCSI_PASSWORD  LoginTimeout=15

Save and close the file.

Start the iscsi service

Type the following command to start iscsi service so that you can see block device:
# chkconfig iscsi on # /etc/init.d/iscsi start
Run any one of the following to find out new block device name:
# fdisk -l # tail -f /var/log/messages # find /sys/devices/platform/host* -name "block*"

Format iSCSI device

Use fdisk and mkfs.ext3 commands. First, create a partition (assuming that /dev/sdc is a new block device assigned to iscsi) :
# fdisk /dev/sdc # mkfs.ext3 /dev/sdc1
Create /mnt/iscsi directory:
# mkdir -p /mnt/iscsi
Open /etc/fstab file and append config directive:
/dev/sdc1 /mnt/iscsi ext3 _netdev 0 0
Save and close the file. Mount the parition /dev/sdc1:
# mount -a # df -H

Continue reading rest of the Linux iSCSI Storage Configuration series.

CentOS / Red Hat Linux: Install and manage iSCSI Volume
CentOS 4 / Redhat Enterprise Linux 4 iSCSI Installation and Configuration ( iscsi initiators )
How To: Windows Server / XP / 2000 / 2003 ISCSI Initiator Configuration

Next in series » »

« « Previous in series

UNIX Find A File Command

Posted: 14 May 2008 08:14 PM CDT

I've just started to use Sun Solaris UNIX in our Labs. I'd like to know more about finding files from from the shell prompt. How do I find a file under UNIX?

Answer to "UNIX Find A File Command"