Ubuntu 20.04 + btrfs + mirrored drives

I run a Linux machine at home. It’s mostly a server, although sometimes I use it as a desktop. I use it for SSH, samba, and I have apache hosting family photos. The photos sit on a secondary drive, and lately it manifest sector errors. Some of photos were no longer accessible.

Fortunately, I have a backup on an external hard drive. Thank goodness for backups!

I decided that it was time to upgrade to Ubuntu 20.04. This time, I wanted mirrored drives with a file system that detects silent data corruption 2.

I started with btrfs, partitioned my first drive and installed Ubuntu. But wait? Should I have chosen ZFS?

ZFS is a first class citizen in Ubuntu, and is reportedly rock solid. The documentation is good.

In contrast, it takes some searching to find good btrfs documentation, and some of the documentation refers to scary features that might eat your data — not comforting. Yet Facebook uses btrfs, so that’s good, right? On the other hand, they have people whose job it is to use it for the right use cases, and configure it appropriately. I don’t.

Oh, and there’s bcachefs! It sounds great, and is reportedly faster, more modern than btrfs or ZFS, and more reliable that btrfs. Unfortunately, it’s not part of most Linux distributions yet, nor is it feature complete. I’ll look forward to using in in three to five years.

Ultimately, I chose btrfs because it doesn’t require that I dedicate the entire hard drive, and because I already had some momentum having started my install with it.

I found the following guide to be helpful to set it up btrfs in mirrored mode: https://work-work.work/blog/2018/12/01/ubuntu-1804-btrfs.html

Arch has good documentation for btfs: https://wiki.archlinux.org/index.php/Btrfs

Once installed, it took hours to add missing packages, reconfigure software and restore from backup. Here’s some of what I installed:

sudo apt-get install meld screen jhead socat apache2 zsh tree tcpdump zip xz-utils zsh vlc ffmpeg samba fail2ban mutt curl strace git ruby rsync python3-virtualenv fetchmail procmail postfix openssh-server netcat-openbsd lsof gwenview dnsutils dosfstools build-essential htop silversearcher-ag mailutils vim-gtk3 at fetchmail

Simulate dropped or latent packets in Linux

I’m documenting this more for my own reference than anything, partly because I’ve used ‘tc’ off and on over the years.

  • https://stackoverflow.com/questions/614795/simulate-delayed-and-dropped-packets-on-linux
  • https://wiki.linuxfoundation.org/networking/netem

    tc qdisc add dev tun0 root netem loss 30%
    tc qdisc show dev tun0
    tc qdisc change dev tun0 root netem loss 0.1%

    When finished:

    tc qdisc del dev tun0 root
  • Ubuntu on Windows: Refreshing & Fast

    Microsoft has been doing interesting things with Windows, such as adding Windows Subsystem for Linux, which allowed me to install and run Ubuntu from the Windows app store.

    I love having a full and familiar Linux bash command shell at my fingertips, with the utilities I know and love, including ssh. It’s fast.


    Prior to this, I used cygwin on Windows, which was also good. However, I prefer Ubuntu, mostly.

    There are some caveats:

    • The home directory is in a different place from the Windows home. So for easy access, I symlink Documents, Downloads, Pictures and Videos to my Ubuntu homedir.
    • Still need to keep the Ubuntu software up-to-date. Microsoft’s app store doesn’t do it for you. Run sudo apt-get update ; sudo apt-get upgrade
    • Removable drives aren’t mapped into the filesystem automatically — cygwin was better in this regard.
    • It’s a subsystem — a container, so it doesn’t manage Windows. E.g.
      — Can’t reboot from the command line
      — Can’t manage Windows processes or users

    Screen brightness: Linux + Lenovo P50

    I run Ubuntu on my Lenovo P50, and the backlight keys haven’t ever worked. Here’s how I got it working.

    sudo apt install xbacklight

    Then I mapped the keys using Settings > Devices > Keyboard and added mappings for the following:

    Windows-F5: xbacklight -inc 5 -time 1 -steps 1
    Windows-F6: xbacklight -dec 5 -time 0 -steps 1

    Lightning Memory Mapped Database

    My Ubuntu box decided to update this library today, so I learned about a very fast and cool key-value database. It replaces BerkleyDB on Linux systems and is also used to persist data for Redis, InfluxDB, and has even been adapted for use with SQLite — called SQLightning (20x faster).


    Ubuntu 18.04 & Gnome 3

    I’ve upgraded three computers to Ubuntu 18.04. Although I appreciate the modern software (including LibreOffice), each upgrade has had different issues.

    Lenovo Server: upgrade was rocky because the root partition ran out of space part way through the upgrade. I hand-recovered and managed to get it to finish. Later, the journal (systemd journal) went nuts and filled up my root partition (which is shared with /var) with log messages — causing so much I/O that it was quite slow to log in to my computer. Once I figured out how to vacuum the journal, I recovered space, and set the journal size smaller. Now it seems to be working well.

    System76 Galego Ultrapro: upgraded without a hitch. However, power management is less-than stellar. It won’t go to sleep when I want it to, and it comes out of sleep when I don’t want it to. Update: Later updates fixed the problem.

    Lenovo P50 with NVidia graphics card: It worked better at driving two external monitors with Ubuntu 16.04. It mostly works with 18.04, but it’s more temperamental. The upgrade didn’t go smoothly, aborted early, and I had to hand-recover, which, fortunately, worked out. I needed a new version of VMWare Workstation. The screen brightness buttons don’t work, even after trying various proposed solutions.

    Later, I found out that I was missing a package that allowed me to mount external encrypted drives. This post had a solution: https://github.com/pop-os/pop/issues/163

    sudo apt install libblockdev-crypto2
    systemctl restart udisks2.service

    Things I appreciate about Gnome 3 (Ubuntu 18.04):

    • Keyboard shortcuts, including WINDOWS + left-click-window + drag
    • Window snapping: WINDOWS-LEFT, WINDOWS-RIGHT, etc. Very similar to Windows
    • High-DPI support works well, which is excellent for my Lenovo P50 with a 4K display (4K is too much resolution for a laptop screen, but it was the only option with the Xeon processors).
    • Looks great

    Things I dislike about Gnome 3 (Ubuntu 18.04):

    • Clock doesn’t include day of month by default. Requires gnome-tweak tool to enable. Sloppy and difficult.
    • Too many clicks to get to network settings, including VPN. It used to be easier.
    • Can’t share my connection with wired-via-USB-cable computers anymore. Th reported workaround, which doesn’t work at all for me: launch nm-connection-editor.
    • Login screen shows a background instead of a list of users, until I press a button or swipe. Please don’t follow Windows here. It’s dumb.
    • When I zoom in on a folder in Nautilus, it zooms all other folders, including my desktop icons.

    Other things I dislike with Gnome — longstanding issues that existed before Gnome 3:

    • Nautilus uses too much white space between images when zooming in on icon view. It should be proportional — like windows Explorer does. I.e. when the images are 0.5×0.5 inches, it’s fine to have 0.5 inches between icons. But when the icons are 3″x3″, I don’t want or need 3″ of white space between icons! (This isn’t an issue with Gnome 3 — it’s a long-standing issue with Nautilus)
    • Nautilus doesn’t show image meta-data such as camera model for images — I like to sort by camera model.
    • Lack of a photo screensaver. I live without it, but it still frustrates me that Gnome is the only desktop, which, by default, doesn’t include one. Windows, Mac and KDE are much better in this regard.

    I love using Linux, but Windows is squarely better at some things.

    Add a camera via WPS to a LEDE/OpenWRT router

    I have some WiFi cameras that can be added to a router via WPS. Here’s how I got it to work with one of my LEDE routers. On the other one, somehow, I broke its ability to do WiFi completely, so this can be dangerous — I had to re-install LEDE. YMMV.

    OpenWRT/LEDE Instructions:

    First, backup the router config — always a good idea!


    opkg update
    opkg remove wpad-mini
    opkg install wpad hostapd-utils
    opkg upgrade dnsmasq
    cp /etc/config/wireless /etc/config/wireless.orig
    vi /etc/config/wireless and change wps_pushbutton to '1' -- but only for one interface.

    Check to see if WiFi is working. If not, use the ethernet port connected to a laptop to log back in, and update the firmware that isn’t broken. There may be a better way, but that’s worked for me.

    Put the router into WPS mode (note: this times out after a while):

    hostapd_cli wps_pbc

    Other instructions say to run this (YMMV):

    hostapd_cli -i wlan1 wps_pbc

    Within a minute or so, push the WPS mode button on the camera.

    Yubikey 4 GPG key generation (Ubuntu)

    Install supporting software

    sudo apt-add-repository ppa:yubico/stable
    sudo apt-get update
    sudo apt-get install scdaemon -y
    sudo apt-get install python-setuptools python-crypto python-pyscard python-pyside pyside-tools libykpers-1-1 pcscd -y
    sudo apt-get install yubioath-desktop yubikey-personalization yubikey-personalization-gui yubikey-manager  -y

    Insert Yubikey and Generate key

    gpg --card-edit
    gpg/card> admin
    gpg/card> generate
    gpg/card> quit

    export and backup the public keys, because the Yubikey only stores the private portion of the key

    gpg --armor --export $KEYID > mykey.pub

    Require touching the Yubikey button to authenticate, sign, or encrypt:

    ykman openpgp touch aut on 
    ykman openpgp touch sig on 
    ykman openpgp touch enc on 

    Change the pin

    gpg --card-edit
    gpg/card> admin
    gpg/card> passwd
    gpg/card> quit

    Change yubikey information

    gpg --card-edit
    gpg/card> name
    gpg/card> lang
    gpg/card> quit


    LEDE awesomeness

    I’ve had what I thought was a great WiFi router for the past 3 years. The vendor continues to provide firmware updates, which is admirable.

    Having heard of the awesome improvements that are being made by folks in the LEDE fork of OpenWRT (in the area of eliminating bufferbloat), I thought it was time for an upgrade. So I purchased an Archer C7 version 2 router, and today, I installed LEDE. Installation was a breeze. Configuring LEDE isn’t as easy as most consumer WiFi routers, but the payoff has been good.

    My downstream 2GHz WiFi cameras and networking gear seem to be staying online better, and streaming live video works better as well. I’m not sure if my family notices much of a difference, but I do. I appreciate the folks who have brought me better networking.