LEDE awesomeness

I’ve had what I thought was a great WiFi router for the past 3 years. The vendor continues to provide firmware updates, which is admirable.

Having heard of the awesome improvements that are being made by folks in the LEDE fork of OpenWRT (in the area of eliminating bufferbloat), I thought it was time for an upgrade. So I purchased an Archer C7 version 2 router, and today, I installed LEDE. Installation was a breeze. Configuring LEDE isn’t as easy as most consumer WiFi routers, but the payoff has been good.

My downstream 2GHz WiFi cameras and networking gear seem to be staying online better, and streaming live video works better as well. I’m not sure if my family notices much of a difference, but I do. I appreciate the folks who have brought me better networking.

UPnP, SSDP, mDNS, LLMNR, etc. on the home network

Sometime in the distant past, I was aware of Universal Plug and Play (UPnP), but I didn’t know much about it. It’s a technology that allows devices in the home to talk to each other without prior configuration — it allows auto-discovery and configuration of printers and media servers, among other things.

The auto-discovery happens via SSDP (Simple Service Discovery Protocol). A device joins a network and announces “I’m here!”, and then other device can choose to respond. Even if the device gets a different IP address, it can still be uniquely identified by its unique identifier (UUID).

Here’s more information about UPnP and related protocols that run on the home network:



UPnP protocol (no authentication):

  • Discovery (SSDP)
  • Description – HTTPU and HTTPMU
  • Control
  • Event notification
  • Presentation

UPnP has well defined device profiles for:

  • Audio & Video — DLNA, and
  • Routers:

    • Internet Gateway Device Protocol

    • Retrieve external IP addr

    • Enumerate port mappings
    • Add/Remove port mappings & port forwarding: firewall-hole-punching
  • Devices Profile for Web Services (DPWS)

Other protocols that help on the home network:

  • LLMNR: Link-local Multicast Name Resolution — implemented by Microsoft in Windows.
  • mDNS (multicast DNS) runs on port 5353. Uses .local hostnames.
  • DNS-SD: DNS service discovery. Can use DNS or mDNS.

Apple’s Bonjour uses mDNS and DNS-SD. Linux’s Avahi uses IPv4LL, mDNS, and DNS-SD. Linux’s systemd has “systemd-resolve”, a command-linetool to resolve hostnames on a network via DNS, mDNS, and LMMNR.

Runtime debugging tools for Linux

Here’s a useful presentation on Linux debugging tools — tools that don’t require source code, additional prints or logging.


  • strace has a new flag that I didn’t know about: -y, which prints the paths that are associated with file descriptors.

  • opensnoop lets you see the details of open() calls across the entire system, or for an individual process, or for paths containing certain characters, or it can print the file paths that couldn’t be opened.

  • pgrep shows the stack trace of a running process, which can be useful to get an idea of what a program spends most of its time doing.

  • dstat shows system resource stats. It is a replacement for vmstat, iostat and ifstat.

  • htop — a more beautiful ‘top’, and easier to use. I still mostly use ‘top’ because it is installed by default. Other great tools I use include ‘powertop’ and ‘iotop’.

  • ngrep — an alternative to tcpdump, but allows the use of regexes to match plain-text data in packets.

  • tcpdump — useful when troubleshooting network connections between servers.

  • wireshark — a more UI-friendly tool than tcpdump, with dissectors for most protocols

Chrome Remote Desktop

I needed to help a friend on a remote computer recently. A coworker told me about Chrome Remote Desktop, which works on any computer that has a Chrome browser, including Linux, Mac, Windows, iPhone and Android.

Chrome Remote Desktop is an easy-to-install plugin for Chrome, and is gratis (no cost). It worked quite well, and I’m happy to recommend it.

Alternatives include copilot.com, which is free on weekends. Lifehacker has a list of solutions as well.

How to store passwords: Use Argon2

If you’re designing a service that requires passwords for authentication, store them using the Argon2 or bcrypt password hashing functions. Don’t use MD5, SHA-1, SHA-2 or SHA-3 — they’re not designed to keep passwords secure against attackers that gain access to your password database.

Reference article: How LinkedIn’s password sloppiness hurts us all by Jeremi M. Gosney

If [online services] aren’t using something like bcrypt or Argon2 for password storage, then they’re doing things very, very wrong. But slow hashing is no longer as effective of a solution as it could have once been had it only been adopted sooner.

When you suspect a password database has been compromised, even just in part, you cash in on that insurance policy [of using forced password resets] immediately by activating your incident response team and your public relations team.

What is Argon2? It’s the winning algorithm from the Password Hashing Competition. Argon2 has been added to recent versions of libsodium.


At work, we use git, and git supports hooks, including pre-commit hooks. Rather than write my own, and do it poorly, I’m using a tool called pre-commit, created by engineers at Yelp.com. To them, I offer my thanks.

Containerization – the beginning of a long journey

I read this today, and thought it’s worth sharing:

The impact of containerization in redefining the enterprise OS is still vastly underestimated by most; it is a departure from the traditional model of a single-instance, monolithic, UNIX user space in favor of a multi-instance, multi-version environment using containers and aggregate packaging. We are talking about nothing less than changing some of the core paradigms on which the software industry has been working for the last 20 – if not 40 – years.”

And yet it is tempered with reality:

we are really only at the beginning of a long journey…”


Grepping archived, rotated log files — in order

Say you’ve got the following log files with the oldest entries in myapi.log.3.gz:


If you want to ‘grep’ them for a string, in order of date, oldest to newest, there’s no need to extract them one at a time, and there’s no need to concatenate the files first. Use sort to put the files in the proper order, and zgrep to search though the compressed files.

Here’s how to order the file list:

ls myapi.log.*.gz | sort -nr -t . -k 3,3

Here’s how to ‘zgrep’ them in the proper order:

ls myapi.log.*.gz | sort -nr -t . -k 3,3 | xargs zgrep “404”

SaltStack RPM archive

SaltStack is a great infrastructure management tool. I don’t always want to use the latest version, so it’s useful to know where to download older releases. The problem is that using EPEL to install it on CentOS/RHEL systems only offers the latest version.

Fortunately, it’s possible to download old RPM packages from the following URLs — thanks to “forrest” on the SaltStack IRC channel for digging up this information.




Here’s how I’ve upgrade my salt infrastructure:

– Make a backup of the salt-master
– Upgrade the salt master
– Install the ‘at’ package on all minions (my minions are linux)
– salt ‘*’ pkg.install at
– Enable and start the at service
– salt ‘*’ service.enable atd
– salt ‘*’ service.start atd
– Run a script
– Copy and paste the code below into your salt directory, typically /srv/salt/upgrade_minion.sh
– salt ‘*’ cmd.script salt://upgrade_minion.sh timeout=15

Here’s the script. Note: It would be safer (and faster) to download from a trusted internal HTTP server.

VERSION=$(salt-minion –version | awk ‘{print $2}’)

cd /root
if [ X$VERSION != ‘2014.1.7’ ] ; then
echo “upgrading `date`”
set -e
curl https://kojipkgs.fedoraproject.org/packages/salt/2014.1.7/3.el6/noarch/salt-2014.1.7-3.el6.noarch.rpm -o salt-2014.1.7-3.el6.noarch.rpm
curl https://kojipkgs.fedoraproject.org/packages/salt/2014.1.7/3.el6/noarch/salt-minion-2014.1.7-3.el6.noarch.rpm -o salt-minion-2014.1.7-3.el6.noarch.rpm
sha1sum salt-2014.1.7-3.el6.noarch.rpm
sha1sum salt-minion-2014.1.7-3.el6.noarch.rpm
echo ‘/etc/init.d/salt-minion stop ; /bin/rm -rf /var/cache/salt ; truncate -s 0 /var/log/salt/minion ; rpm -Uhv /root/salt*2014.1.7-3*rpm ; /etc/init.d/salt-minion start’ | at now
echo “Already upgraded to $VERSION”

Introduction to SaltStack

I had the opportunity to give an introduction to SaltStack at the OpenWest conference this week (Video on YouTube).

As I was setting up for the presentation, I realized I needed an HDMI cable, and there wasn’t one in the room. A conference helper found a cable at the last minute, and we started on time (thank you!).

From what the audience said, many people are interested in using SaltStack, but haven’t started yet. It’s a fantastic tool for remote execution and configuration management. The documentation is initially a bit intimidating, but it’s easier to get started than it appears — especially using one of he many online tutorials. And there are many examples available from saltstarters.org.

SaltStack releases for CentOS/RHEL have become more stable over the past 10 months, and I’m excited about it’s future. It has a vibrant development community, and they’re implementing cool new features to extend its capabilities, including:

– RAET, an optional new, secure UDP communications protocol, which will easily support more than 10,000 minions. https://github.com/saltstack/raet
– Proxy minions will eventually allow the configuration of load balancers and networking equipment.

And advanced features, already present, sound like they could be useful:

– The orchestration framework, based around the reactor, can allow configuration when certain events are triggered. For example, Jenkins, upon building successfully, could send an event from the Salt minion to the master, and the Salt Master could automatically deploy the build to a test environment.
– Custom module, state and proxy support. In /srv/salt, create a _modules directory, etc. They can either override the existing module, or add new ones.