amuck-landowner

[Tutorial] SSD Cache with Flashcache

N

Nett

Article Submitter
Verified Provider
Having SSD cache set up on a HDD based server is an inexpensive way of increasing IO throughput and system performance. It does not require any special hardware such as a RAID card to do the job.

Flashcache is an open source project maintained by Facebook. It is built on top of the Linux kernel's device mapper and runs as a kernel module.

In this tutorial, I'll be demonstrating the set up on a OpenVZ VPS node. This tutorial can also be used to set up SSD cache on many other types of servers, cPanel is one example.

The server used for demonstration purposes has the following specifications:
Dual L5520 CPU
24GB RAM
2x 1TB HDD in Software RAID 1
1x 128GB SSD for caching

Spoiler




[root@vpsboard-tutorial ~]# grep 'model name' /proc/cpuinfo | head -n1
model name : Intel® Xeon® CPU L5520 @ 2.27GHz
[root@vpsboard-tutorial ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/md1 15G 2.3G 12G 17% /
tmpfs 12G 0 12G 0% /dev/shm
/dev/md0 477M 89M 364M 20% /boot
/dev/md3 894G 7.5G 841G 1% /vz
[root@vpsboard-tutorial ~]# free -m
total used free shared buffers cached
Mem: 23967 4054 19912 0 108 1306
-/+ buffers/cache: 2639 21327
Swap: 8183 0 8183
[root@vpsboard-tutorial ~]# uname -r
2.6.32-042stab106.4

[root@vpsboard-tutorial ~]# cat /etc/redhat-release
CentOS release 6.6 (Final)
 


  • Before we start everything, please install OpenVZ and any VPS control panel (if required). Make sure that your server is partitioned according to OpenVZ's guidelines available on their website. If you are using cPanel, please make sure /home is on a separate partition.

    Reboot the server before installing Flashcache.
  • First of all, note the disk IO without SSD cache:

    [root@vpsboard-tutorial ~]# dd if=/dev/zero of=/vz/test bs=64k count=16k conv=fdatasync; rm -f /vz/test
    16384+0 records in
    16384+0 records out
    1073741824 bytes (1.1 GB) copied, 10.2626 s, 105 MB/s
  • Next, we are going to install all required packages:
     
    Code: 
    yum -y install wget make gcc nano git yum-utils kernel-devel dkms



    Spoiler




    [root@vpsboard-tutorial /]# yum -y install wget make gcc nano git yum-utils kernel-devel dkms
    Loaded plugins: fastestmirror, security
    Setting up Install Process
    Loading mirror speeds from cached hostfile
    * base: linux.mirrors.es.net
    * epel: linux.mirrors.es.net
    * extras: mirror.pac-12.org
    * openvz-kernel-rhel6: mirror.supremebytes.com
    * openvz-utils: mirror.supremebytes.com
    * soluslabs: mirror.us1.soluslabs.net
    * updates: mirror.supremebytes.com
    Package wget-1.12-5.el6_6.1.x86_64 already installed and latest version
    Package 1:make-3.81-20.el6.x86_64 already installed and latest version
    Package gcc-4.4.7-11.el6.x86_64 already installed and latest version
    Package nano-2.0.9-7.el6.x86_64 already installed and latest version
    Package yum-utils-1.1.30-30.el6.noarch already installed and latest version
    Resolving Dependencies
    --> Running transaction check
    ---> Package dkms.noarch 0:2.2.0.3-30.git.7c3e7c5.el6 will be installed
    ---> Package git.x86_64 0:1.7.1-3.el6_4.1 will be installed
    --> Processing Dependency: perl-Git = 1.7.1-3.el6_4.1 for package: git-1.7.1-3.el6_4.1.x86_64
    --> Processing Dependency: perl(Git) for package: git-1.7.1-3.el6_4.1.x86_64
    --> Processing Dependency: perl(Error) for package: git-1.7.1-3.el6_4.1.x86_64
    ---> Package kernel-devel.x86_64 0:2.6.32-504.12.2.el6 will be installed
    --> Running transaction check
    ---> Package perl-Error.noarch 1:0.17015-4.el6 will be installed
    ---> Package perl-Git.noarch 0:1.7.1-3.el6_4.1 will be installed
    --> Finished Dependency Resolution

    Dependencies Resolved

    =====================================================================================================================================
    Package Arch Version Repository Size
    =====================================================================================================================================
    Installing:
    dkms noarch 2.2.0.3-30.git.7c3e7c5.el6 epel 77 k
    git x86_64 1.7.1-3.el6_4.1 base 4.6 M
    kernel-devel x86_64 2.6.32-504.12.2.el6 updates 9.4 M
    Installing for dependencies:
    perl-Error noarch 1:0.17015-4.el6 base 29 k
    perl-Git noarch 1.7.1-3.el6_4.1 base 28 k

    Transaction Summary
    =====================================================================================================================================
    Install 5 Package(s)

    Total download size: 14 M
    Installed size: 39 M
    Downloading Packages:
    (1/5): dkms-2.2.0.3-30.git.7c3e7c5.el6.noarch.rpm | 77 kB 00:00
    (2/5): git-1.7.1-3.el6_4.1.x86_64.rpm | 4.6 MB 00:00
    (3/5): kernel-devel-2.6.32-504.12.2.el6.x86_64.rpm | 9.4 MB 00:00
    (4/5): perl-Error-0.17015-4.el6.noarch.rpm | 29 kB 00:00
    (5/5): perl-Git-1.7.1-3.el6_4.1.noarch.rpm | 28 kB 00:00
    -------------------------------------------------------------------------------------------------------------------------------------
    Total 15 MB/s | 14 MB 00:00
    Running rpm_check_debug
    Running Transaction Test
    Transaction Test Succeeded
    Running Transaction
    Installing : 1:perl-Error-0.17015-4.el6.noarch 1/5
    Installing : perl-Git-1.7.1-3.el6_4.1.noarch 2/5
    Installing : git-1.7.1-3.el6_4.1.x86_64 3/5
    Installing : kernel-devel-2.6.32-504.12.2.el6.x86_64 4/5
    Installing : dkms-2.2.0.3-30.git.7c3e7c5.el6.noarch 5/5
    Verifying : git-1.7.1-3.el6_4.1.x86_64 1/5
    Verifying : 1:perl-Error-0.17015-4.el6.noarch 2/5
    Verifying : perl-Git-1.7.1-3.el6_4.1.noarch 3/5
    Verifying : dkms-2.2.0.3-30.git.7c3e7c5.el6.noarch 4/5
    Verifying : kernel-devel-2.6.32-504.12.2.el6.x86_64 5/5

    Installed:
    dkms.noarch 0:2.2.0.3-30.git.7c3e7c5.el6 git.x86_64 0:1.7.1-3.el6_4.1 kernel-devel.x86_64 0:2.6.32-504.12.2.el6

    Dependency Installed:
    perl-Error.noarch 1:0.17015-4.el6 perl-Git.noarch 0:1.7.1-3.el6_4.1

    Complete!

  • Install vzkernel-devel (SKIP this step if you are not running OpenVZ)

    This demo server runs 2.6.32-042stab106.4 kernel. Please make sure that you install the kernel matching uname -r output.

    wget http://download.openvz.org/kernel/branches/rhel6-2.6.32/042stab106.4/vzkernel-devel-2.6.32-042stab106.4.x86_64.rpm
    yum -y install vzkernel-devel-2.6.32-042stab106.4.x86_64.rpm



    Spoiler




    [root@vpsboard-tutorial /]# wget http://download.openvz.org/kernel/branches/rhel6-2.6.32/042stab106.4/vzkernel-devel-2.6.32-042stab106.4.x86_64.rpm
    --2015-04-19 03:41:08-- http://download.openvz.org/kernel/branches/rhel6-2.6.32/042stab106.4/vzkernel-devel-2.6.32-042stab106.4.x86_64.rpm
    Resolving download.openvz.org... 199.115.104.11, 2620:e6::104:11
    Connecting to download.openvz.org|199.115.104.11|:80... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 9396333 (9.0M) [application/x-rpm]
    Saving to: `vzkernel-devel-2.6.32-042stab106.4.x86_64.rpm'

    100%[===========================================================================================>] 9,396,333 609K/s in 14s

    2015-04-19 03:41:22 (637 KB/s) - `vzkernel-devel-2.6.32-042stab106.4.x86_64.rpm' saved [9396333/9396333]

    [root@vpsboard-tutorial /]# yum -y install vzkernel-devel-2.6.32-042stab106.4.x86_64.rpm
    Loaded plugins: fastestmirror, security
    Setting up Install Process
    Examining vzkernel-devel-2.6.32-042stab106.4.x86_64.rpm: vzkernel-devel-2.6.32-042stab106.4.x86_64
    Marking vzkernel-devel-2.6.32-042stab106.4.x86_64.rpm to be installed
    Loading mirror speeds from cached hostfile
    * base: linux.mirrors.es.net
    * epel: linux.mirrors.es.net
    * extras: mirror.pac-12.org
    * openvz-kernel-rhel6: mirror.supremebytes.com
    * openvz-utils: mirror.supremebytes.com
    * soluslabs: mirror.us1.soluslabs.net
    * updates: mirror.supremebytes.com
    Resolving Dependencies
    --> Running transaction check
    ---> Package vzkernel-devel.x86_64 0:2.6.32-042stab106.4 will be installed
    --> Finished Dependency Resolution

    Dependencies Resolved

    =====================================================================================================================================
    Package Arch Version Repository Size
    =====================================================================================================================================
    Installing:
    vzkernel-devel x86_64 2.6.32-042stab106.4 /vzkernel-devel-2.6.32-042stab106.4.x86_64 25 M

    Transaction Summary
    =====================================================================================================================================
    Install 1 Package(s)

    Total size: 25 M
    Installed size: 25 M
    Downloading Packages:
    Running rpm_check_debug
    Running Transaction Test
    Transaction Test Succeeded
    Running Transaction
    Installing : vzkernel-devel-2.6.32-042stab106.4.x86_64 1/1
    Verifying : vzkernel-devel-2.6.32-042stab106.4.x86_64 1/1

    Installed:
    vzkernel-devel.x86_64 0:2.6.32-042stab106.4

    Complete!

  • Install Flashcache

    Please make sure that you compile Flashcache for the kernel matching uname -r output.
     

    git clone https://github.com/facebook/flashcache.git
    cd flashcache
    make KERNEL_TREE=/usr/src/kernels/2.6.32-042stab106.4
    make install KERNEL_TREE=/usr/src/kernels/2.6.32-042stab106.4



    Spoiler




    [root@vpsboard-tutorial /]# git clone https://github.com/facebook/flashcache.git
    Initialized empty Git repository in /flashcache/.git/
    remote: Counting objects: 1295, done.
    remote: Total 1295 (delta 0), reused 0 (delta 0), pack-reused 1295
    Receiving objects: 100% (1295/1295), 569.83 KiB, done.
    Resolving deltas: 100% (783/783), done.
    [root@vpsboard-tutorial /]# cd flashcache
    [root@vpsboard-tutorial flashcache]# make KERNEL_TREE=/usr/src/kernels/2.6.32-042stab106.4
    make -C src KERNEL_TREE=/usr/src/kernels/2.6.32-042stab106.4 PWD=/flashcache/src all
    make[1]: Entering directory `/flashcache/src'
    make -C /usr/src/kernels/2.6.32-042stab106.4 M=/flashcache/src modules V=0
    make[2]: Entering directory `/usr/src/kernels/2.6.32-042stab106.4'
    CC [M] /flashcache/src/flashcache_conf.o
    CC [M] /flashcache/src/flashcache_main.o
    CC [M] /flashcache/src/flashcache_subr.o
    CC [M] /flashcache/src/flashcache_ioctl.o
    CC [M] /flashcache/src/flashcache_procfs.o
    CC [M] /flashcache/src/flashcache_reclaim.o
    CC [M] /flashcache/src/flashcache_kcopy.o
    LD [M] /flashcache/src/flashcache.o
    Building modules, stage 2.
    MODPOST 1 modules
    CC /flashcache/src/flashcache.mod.o
    LD [M] /flashcache/src/flashcache.ko.unsigned
    NO SIGN [M] /flashcache/src/flashcache.ko
    make[2]: Leaving directory `/usr/src/kernels/2.6.32-042stab106.4'
    make -C utils all
    make[2]: Entering directory `/flashcache/src/utils'
    cc -c -I.. -I. -DCOMMIT_REV="\"1.0-234-gb166f3b3ff8c\"" -g flashcache_create.c -o flashcache_create.o
    cc flashcache_create.o -o flashcache_create
    cc -c -I.. -I. -DCOMMIT_REV="\"1.0-234-gb166f3b3ff8c\"" -g flashcache_destroy.c -o flashcache_destroy.o
    cc flashcache_destroy.o -o flashcache_destroy
    cc -c -I.. -I. -DCOMMIT_REV="\"1.0-234-gb166f3b3ff8c\"" -g flashcache_load.c -o flashcache_load.o
    cc flashcache_load.o -o flashcache_load
    cc -c -I.. -I. -DCOMMIT_REV="\"1.0-234-gb166f3b3ff8c\"" -g flashcache_setioctl.c -o flashcache_setioctl.o
    cc flashcache_setioctl.o -o flashcache_setioctl
    cc -c -I.. -I. -DCOMMIT_REV="\"1.0-234-gb166f3b3ff8c\"" -g get_agsize.c -o get_agsize.o
    cc get_agsize.o -o get_agsize
    make[2]: Leaving directory `/flashcache/src/utils'
    make[1]: Leaving directory `/flashcache/src'
    [root@vpsboard-tutorial flashcache]# make install KERNEL_TREE=/usr/src/kernels/2.6.32-042stab106.4
    make -C src KERNEL_TREE=/usr/src/kernels/2.6.32-042stab106.4 PWD=/flashcache/src install
    make[1]: Entering directory `/flashcache/src'
    make -C /usr/src/kernels/2.6.32-042stab106.4 M=/flashcache/src modules V=0
    make[2]: Entering directory `/usr/src/kernels/2.6.32-042stab106.4'
    Building modules, stage 2.
    MODPOST 1 modules
    make[2]: Leaving directory `/usr/src/kernels/2.6.32-042stab106.4'
    install -o root -g root -m 0755 -d /lib/modules/2.6.32-042stab106.4/extra/flashcache/
    install -o root -g root -m 0755 flashcache.ko /lib/modules/2.6.32-042stab106.4/extra/flashcache/
    depmod -a 2.6.32-042stab106.4
    make -C utils all
    make[2]: Entering directory `/flashcache/src/utils'
    make[2]: Nothing to be done for `all'.
    make[2]: Leaving directory `/flashcache/src/utils'
    make -C utils install
    make[2]: Entering directory `/flashcache/src/utils'
    install -d -m 755 /sbin/
    install -m 755 flashcache_create flashcache_destroy flashcache_load flashcache_setioctl get_agsize /sbin/
    make[2]: Leaving directory `/flashcache/src/utils'
    make -C ocf install
    make[2]: Entering directory `/flashcache/src/ocf'
    install -d -m 755 /usr/lib/ocf/resource.d/flashcache
    install -m 755 flashcache /usr/lib/ocf/resource.d/flashcache
    make[2]: Leaving directory `/flashcache/src/ocf'
    make[1]: Leaving directory `/flashcache/src'

  • Flashcache is now installed, load the Flashcache module and make sure it is working:

     

    modprobe flashcache
    lsmod | grep flashcache
    You should get something like this:



    Spoiler




    [root@vpsboard-tutorial flashcache]# modprobe flashcache
    [root@vpsboard-tutorial flashcache]# lsmod | grep flashcache
    flashcache 97695 0
    dm_mod 95654 3 flashcache,dm_mirror,dm_log



    If Flashcache does not show up in the second command run `dmesg | tail` for some error outputs.
  • Unmount the partition to be 'cached' and create a Flashcache volume

    Unmount the partition and ensure it does not show up in `df`.
     

    umount /vz
    Find out the partition's UUID (Universally unique identifier).
     
    Code: 
    grep '/vz' /etc/fstab | cut -d= -f2 | cut -d' ' -f1
    Comment out or remove the line containing the partition from /etc/fstab
     
    Code: 
    nano /etc/fstab
    Create the Flashcache volume
    Here ssd_cache is the name of the volume that we are going to creater and /dev/sdc is the SSD. Replace <uuid> with the actual disk UUID.
    Code: 
    flashcache_create -p back ssd_cache /dev/sdc /dev/disk/by-uuid/<uuid>



    Spoiler




    [root@vpsboard-tutorial /]# umount /vz
    [root@vpsboard-tutorial /]# grep '/vz' /etc/fstab | cut -d= -f2 | cut -d' ' -f1
    7c008696-ccb6-4895-9251-eb230c7f7eb2
    [root@vpsboard-tutorial /]# nano /etc/fstab
    [root@vpsboard-tutorial /]# flashcache_create -p back ssd_cache /dev/sdc /dev/disk/by-uuid/7c008696-ccb6-4895-9251-eb230c7f7eb2
    cachedev ssd_cache, ssd_devname /dev/sdc, disk_devname /dev/disk/by-uuid/7c008696-ccb6-4895-9251-eb230c7f7eb2 cache mode WRITE_BACK
    block_size 8, md_block_size 8, cache_size 0
    Flashcache metadata will use 655MB of your 23967MB main memory

  • Configure Flashcache to automatically start on server startup:
     

    cp /flashcache/utils/flashcache /etc/init.d/
    chmod +x /etc/init.d/flashcache
    chkconfig flashcache on
    nano /etc/init.d/flashcache
    Edit the following to match what we have just done

     
    Code: 
    SSD_DISK=/dev/sdc
BACKEND_DISK=/dev/disk/by-uuid/<uuid>
CACHEDEV_NAME=ssd_cache
MOUNTPOINT=/vz
FLASHCACHE_NAME=ssd_cache
    Save the file. Reboot your server to make sure Flashcache can start automatically.
     
    Code: 
    reboot now
  • After Reboot
  • See if the Flashcache volume is mounted
     
    Code: 
    [root@vpsboard-tutorial ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/md1               15G  2.4G   12G  18% /
tmpfs                  12G     0   12G   0% /dev/shm
/dev/md0              477M   89M  364M  20% /boot
/dev/mapper/ssd_cache 894G  7.5G  841G   1% /vz
    If it is not, run

    Code: 
    service flashcache start
  • Perfect, let's test out the IO!
     
    Code: 
    [root@vpsboard-tutorial ~]# dd if=/dev/zero of=/vz/test bs=64k count=16k conv=fdatasync
16384+0 records in
16384+0 records out
1073741824 bytes (1.1 GB) copied, 6.59464 s, 163 MB/s
    Here you have it, a 55% increase in disk IO. In addition, random IOPS have almost doubled with SSD cache.

Additional Information

  • Flashcache has to be re-compiled and installed after kernel upgrades, so frequent kernel upgrades should be avoided
  • Flashcache does not support TRIM at this moment
  • You may lose data on the SSD that is not yet written to the HDD when the server crashes
  • If you are looking for a more reliable and even faster solution, try a RAID card that supports SSD caching or simply go for pure SSD :)



View full article
 
P

perennate

New Member
Verified Provider
Nett said:
You may lose data on the SSD that is not yet written to the HDD when the server crashes
Click to expand...
Hm, flashcache doesn't support reloading SSD cache after reboot? If so, that seems a bit strange, since there's no advantage compared to caching in memory then (i.e. ignoring fsync).
 
Last edited by a moderator:
N

Nett

Article Submitter
Verified Provider
perennate said:
Hm, flashcache doesn't support reloading SSD cache after reboot? If so, that seems a bit strange, since there's no advantage compared to caching in memory then (i.e. ignoring fsync).
Click to expand...
I have not tested it personally and I don't like making assumptions :).

Here is a snippet from the FlashCache documentation:

Node crashes or power failures will not result in data loss, but they will result in the cache losing VALID and non-DIRTY cached blocks.
Click to expand...
 
P

perennate

New Member
Verified Provider
Ah, that makes more sense, the snippet sounds like it just means it'll discard anything that doesn't need to be written back from the cache (i.e., discard everything but dirty blocks).

In other words, you lose your read cache, so it's going to need some time to warm-up, but anything that hasn't been reflected on hard disk yet will be preserved for writeback.
 
Last edited by a moderator:
devonblzx

devonblzx

New Member
Verified Provider
You didn't mention writeback, writethrough, and writearound.

Writeback is the default and fastest, all data is written first to the SSD and then later synced to the disks.  If your SSD dies, this can cause you to lose a lot of data and possibly cause large amounts of data corruption.

Writethrough is safe because all data is  written to the SSD and disks at the same time.  So random writes will be much slower but random reads will still be much improved.

Writearound is mostly read cache.  All writes will be initially written to the disks, when a file is read from the disks, it is also written to the SSD, so it can be read faster the next time.

Some other concerns I had:

Flashcache has to be re-compiled and installed after kernel upgrades, so frequent kernel upgrades should be avoided
Click to expand...
That isn't a good reason to not update your kernel.  Security fixes are released often.  Flashcache has an easy setup with DKMS.  You even had 'dkms' in your yum install...

flashcache_create -p back ssd_cache /dev/sdc /dev/disk/by-uuid/<uuid>
Click to expand...
Why not use UUID for both?  You don't need to use UUID, but if you are concerned about disks coming up differently, then you should use UUID for both.
 
Last edited by a moderator:
devonblzx

devonblzx

New Member
Verified Provider
InertiaNetworks-Ryan said:
Suddenly ZFS's caching looks easier to implement.
Click to expand...
Flashcache is actually very easy to setup compared to other caching technologies (like bcache and dm-cache).

You need to have the EPEL repo for dkms but as far as the install goes, it is this simple:


yum -y install vzkernel-devel dkms
git clone https://github.com/facebook/flashcache.git
cd flashcache
make -f Makefile.dkms

DKMS will keep the module installed for every kernel update.
 
Last edited by a moderator:
N

Nett

Article Submitter
Verified Provider
devonblzx said:
You didn't mention writeback, writethrough, and writearound.

Writeback is the default and fastest, all data is written first to the SSD and then later synced to the disks.  If your SSD dies, this can cause you to lose a lot of data and possibly cause large amounts of data corruption.

Writethrough is safe because all data is  written to the SSD and disks at the same time.  So random writes will be much slower but random reads will still be much improved.

Writearound is mostly read cache.  All writes will be initially written to the disks, when a file is read from the disks, it is also written to the SSD, so it can be read faster the next time.

Some other concerns I had:

That isn't a good reason to not update your kernel.  Security fixes are released often.  Flashcache has an easy setup with DKMS.  You even had 'dkms' in your yum install...

Why not use UUID for both?  You don't need to use UUID, but if you are concerned about disks coming up differently, then you should use UUID for both.
Click to expand...
If you follow this tutorial, the cached volume that you create will be writeback. Flashcache's startup script only supports writeback and that's why I did not talk about the other methods.

Thanks for the suggestions!
 
You must log in or register to reply here.
Top
amuck-landowner