This is a comparison of various block size settings for both squashfs paired with nbd(network block device). I was looking to see if it out-performs standard nfs as well as look into optimal block size settings. It seems default block settings are optimal for squashfs (128k) and nbd(1024). Not only is nbd + squashfs faster but they also reduces the load on the network itself this helps alleviate the current 1000 backbone bottleneck between switches.
Time in seconds – lower is better.
| 512 nbd | 1024 nbd | 2048 nbd | 4096 nbd | Size | |
| 4k squash | 81.27 | 96.00 | 96.05 | 96.37 | 7329.35 |
| 8k squash | 65.46 | 64.77 | 65.29 | 65.36 | 7114.52 |
| 16k squash | 54.75 | 55.55 | 55.66 | 55.65 | 6956.80 |
| 32k squash | 48.04 | 59.56 | 47.65 | 57.51 | 6831.49 |
| 64k squash | 48.01 | 47.94 | 47.79 | 47.80 | 6728.61 |
| 128k squash | 47.86 | 46.78 default | 46.73 | 46.86 | 6676.55 |
CPU Usage – lower is better:
| 4k squash | 33.00% | 28.00% | 28.00% | 27.00% |
| 8k squash | 35.00% | 34.00% | 35.00% | 35.00% |
| 16k squash | 37.00% | 36.00% | 36.00% | 36.00% |
| 32k squash | 38.00% | 31.00% | 38.00% | 32.00% |
| 64k squash | 38.00% | 38.00% | 38.00% | 38.00% |
| 128k squash | 37.00% | 38.00% | 38.00% | 38.00% |
NFS 58 to 70 seconds
NFS CPU 2%
Times were gathered with:
time cat /usr/bin/* /usr/sbin/*
du -sh /usr/bin /usr/sbin
572M /usr/bin
29M /usr/sbin
——–
601M total
I had compressed the entire /usr partition but only ran tests on /usr/sbin and /usr/bin. Here are the compression for mksquashfs defaults @131072 (128K).
Filesystem size 6836785.50 Kbytes (6676.55 Mbytes)
49.93% of uncompressed filesystem size (13691786.46 Kbytes)
Inode table size 5792060 bytes (5656.31 Kbytes)
27.58% of uncompressed inode table size (20997401 bytes)
Directory table size 5228332 bytes (5105.79 Kbytes)
39.70% of uncompressed directory table size (13169842 bytes)
With ~50% reduction in file-system size I do see a significant reduction on the 1000 network backbone running between switches.
Another benefit to the squashfs+nbd is that the clients are much faster at loading menu’s and icons as the directory scans needed to find the icons perform very well with nbd+squashfs and when compared to the poor performance of directory scans over nfs.
