squashfs + nbd compared to nfs

This is a comparison of various block size settings for both squashfs paired with nbd(network block device).  I was looking to see if it out-performs standard nfs as well as look into optimal block size settings.  It seems default block settings are optimal for squashfs (128k) and nbd(1024).  Not only is nbd + squashfs faster but they also reduces the load on the network itself this helps alleviate the current 1000 backbone bottleneck between switches.

Time in seconds – lower is better.

512 nbd 1024 nbd 2048 nbd 4096 nbd Size
4k squash 81.27 96.00 96.05 96.37 7329.35
8k squash 65.46 64.77 65.29 65.36 7114.52
16k squash 54.75 55.55 55.66 55.65 6956.80
32k squash 48.04 59.56 47.65 57.51 6831.49
64k squash 48.01 47.94 47.79 47.80 6728.61
128k squash 47.86 46.78 default 46.73 46.86 6676.55

 

CPU Usage – lower is better:

4k squash 33.00% 28.00% 28.00% 27.00%
8k squash 35.00% 34.00% 35.00% 35.00%
16k squash 37.00% 36.00% 36.00% 36.00%
32k squash 38.00% 31.00% 38.00% 32.00%
64k squash 38.00% 38.00% 38.00% 38.00%
128k squash 37.00% 38.00% 38.00% 38.00%

NFS  58 to 70 seconds
NFS CPU  2%

 

Times were gathered with:
time cat /usr/bin/* /usr/sbin/*

du -sh /usr/bin /usr/sbin
572M  /usr/bin
29M /usr/sbin
——–
601M total

I had compressed the entire /usr partition but only ran tests on /usr/sbin and /usr/bin.  Here are the compression for mksquashfs defaults @131072 (128K).

Filesystem size 6836785.50 Kbytes (6676.55 Mbytes)
49.93% of uncompressed filesystem size (13691786.46 Kbytes)
Inode table size 5792060 bytes (5656.31 Kbytes)
27.58% of uncompressed inode table size (20997401 bytes)
Directory table size 5228332 bytes (5105.79 Kbytes)
39.70% of uncompressed directory table size (13169842 bytes)

With ~50% reduction in file-system size I do see a significant reduction on the 1000 network backbone running between switches.

Another benefit to the squashfs+nbd is that the clients are much faster at loading menu’s and icons as the directory scans needed to find the icons perform very well with nbd+squashfs and when compared to the poor performance of directory scans over nfs.

 

 

Share
This entry was posted in Linux. Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *