Category: Tools

Posted on

Runtime debugging tools for Linux

Here’s a useful presentation on Linux debugging tools — tools that don’t require source code, additional prints or logging.

http://jvns.ca/blog/2016/09/17/strange-loop-talk/

  • strace has a new flag that I didn’t know about: -y, which prints the paths that are associated with file descriptors.

  • opensnoop lets you see the details of open() calls across the entire system, or for an individual process, or for paths containing certain characters, or it can print the file paths that couldn’t be opened.

  • pgrep shows the stack trace of a running process, which can be useful to get an idea of what a program spends most of its time doing.

  • dstat shows system resource stats. It is a replacement for vmstat, iostat and ifstat.

  • htop — a more beautiful ‘top’, and easier to use. I still mostly use ‘top’ because it is installed by default. Other great tools I use include ‘powertop’ and ‘iotop’.

  • ngrep — an alternative to tcpdump, but allows the use of regexes to match plain-text data in packets.

  • tcpdump — useful when troubleshooting network connections between servers.

  • wireshark — a more UI-friendly tool than tcpdump, with dissectors for most protocols
Posted on

Chrome Remote Desktop

I needed to help a friend on a remote computer recently. A coworker told me about Chrome Remote Desktop, which works on any computer that has a Chrome browser, including Linux, Mac, Windows, iPhone and Android.

Chrome Remote Desktop is an easy-to-install plugin for Chrome, and is gratis (no cost). It worked quite well, and I’m happy to recommend it.

Alternatives include copilot.com, which is free on weekends. Lifehacker has a list of solutions as well.

Posted on

How to store passwords: Use Argon2

If you’re designing a service that requires passwords for authentication, store them using the Argon2 or bcrypt password hashing functions. Don’t use MD5, SHA-1, SHA-2 or SHA-3 — they’re not designed to keep passwords secure against attackers that gain access to your password database.

Reference article: How LinkedIn’s password sloppiness hurts us all by Jeremi M. Gosney

If [online services] aren’t using something like bcrypt or Argon2 for password storage, then they’re doing things very, very wrong. But slow hashing is no longer as effective of a solution as it could have once been had it only been adopted sooner.

When you suspect a password database has been compromised, even just in part, you cash in on that insurance policy [of using forced password resets] immediately by activating your incident response team and your public relations team.

What is Argon2? It’s the winning algorithm from the Password Hashing Competition. Argon2 has been added to recent versions of libsodium.

Posted on

pre-commit

At work, we use git, and git supports hooks, including pre-commit hooks. Rather than write my own, and do it poorly, I’m using a tool called pre-commit, created by engineers at Yelp.com. To them, I offer my thanks.

Posted on

Containerization – the beginning of a long journey

I read this today, and thought it’s worth sharing:

“The impact of containerization in redefining the enterprise OS is still vastly underestimated by most; it is a departure from the traditional model of a single-instance, monolithic, UNIX user space in favor of a multi-instance, multi-version environment using containers and aggregate packaging. We are talking about nothing less than changing some of the core paradigms on which the software industry has been working for the last 20 – if not 40 – years.”

And yet it is tempered with reality:

“…we are really only at the beginning of a long journey…”

http://rhelblog.redhat.com/2015/05/05/rkt-appc-and-docker-a-take-on-the-linux-container-upstream/

Posted on

Grepping archived, rotated log files — in order

Say you’ve got the following log files with the oldest entries in myapi.log.3.gz:

myapi.log.1.gz
myapi.log.2.gz
myapi.log.3.gz

If you want to ‘grep’ them for a string, in order of date, oldest to newest, there’s no need to extract them one at a time, and there’s no need to concatenate the files first. Use sort to put the files in the proper order, and zgrep to search though the compressed files.

Here’s how to order the file list:

ls myapi.log.*.gz | sort -nr -t . -k 3,3

Here’s how to ‘zgrep’ them in the proper order:

ls myapi.log.*.gz | sort -nr -t . -k 3,3 | xargs zgrep “404”

Posted on

SaltStack RPM archive

SaltStack is a great infrastructure management tool. I don’t always want to use the latest version, so it’s useful to know where to download older releases. The problem is that using EPEL to install it on CentOS/RHEL systems only offers the latest version.

Fortunately, it’s possible to download old RPM packages from the following URLs — thanks to “forrest” on the SaltStack IRC channel for digging up this information.

http://koji.fedoraproject.org/koji/packageinfo?packageID=13129

https://kojipkgs.fedoraproject.org/packages/salt/

—-

Here’s how I’ve upgrade my salt infrastructure:

– Make a backup of the salt-master
– Upgrade the salt master
– Install the ‘at’ package on all minions (my minions are linux)
– salt ‘*’ pkg.install at
– Enable and start the at service
– salt ‘*’ service.enable atd
– salt ‘*’ service.start atd
– Run a script
– Copy and paste the code below into your salt directory, typically /srv/salt/upgrade_minion.sh
– salt ‘*’ cmd.script salt://upgrade_minion.sh timeout=15

Here’s the script. Note: It would be safer (and faster) to download from a trusted internal HTTP server.

#!/bin/bash
VERSION=$(salt-minion –version | awk ‘{print $2}’)

cd /root
if [ X$VERSION != ‘2014.1.7’ ] ; then
echo “upgrading `date`”
set -e
curl https://kojipkgs.fedoraproject.org/packages/salt/2014.1.7/3.el6/noarch/salt-2014.1.7-3.el6.noarch.rpm -o salt-2014.1.7-3.el6.noarch.rpm
curl https://kojipkgs.fedoraproject.org/packages/salt/2014.1.7/3.el6/noarch/salt-minion-2014.1.7-3.el6.noarch.rpm -o salt-minion-2014.1.7-3.el6.noarch.rpm
sha1sum salt-2014.1.7-3.el6.noarch.rpm
sha1sum salt-minion-2014.1.7-3.el6.noarch.rpm
echo ‘/etc/init.d/salt-minion stop ; /bin/rm -rf /var/cache/salt ; truncate -s 0 /var/log/salt/minion ; rpm -Uhv /root/salt*2014.1.7-3*rpm ; /etc/init.d/salt-minion start’ | at now
else
echo “Already upgraded to $VERSION”
fi

Posted on

Introduction to SaltStack

I had the opportunity to give an introduction to SaltStack at the OpenWest conference this week (Video on YouTube).

As I was setting up for the presentation, I realized I needed an HDMI cable, and there wasn’t one in the room. A conference helper found a cable at the last minute, and we started on time (thank you!).

From what the audience said, many people are interested in using SaltStack, but haven’t started yet. It’s a fantastic tool for remote execution and configuration management. The documentation is initially a bit intimidating, but it’s easier to get started than it appears — especially using one of he many online tutorials. And there are many examples available from saltstarters.org.

SaltStack releases for CentOS/RHEL have become more stable over the past 10 months, and I’m excited about it’s future. It has a vibrant development community, and they’re implementing cool new features to extend its capabilities, including:

– RAET, an optional new, secure UDP communications protocol, which will easily support more than 10,000 minions. https://github.com/saltstack/raet
– Proxy minions will eventually allow the configuration of load balancers and networking equipment.

And advanced features, already present, sound like they could be useful:

– The orchestration framework, based around the reactor, can allow configuration when certain events are triggered. For example, Jenkins, upon building successfully, could send an event from the Salt minion to the master, and the Salt Master could automatically deploy the build to a test environment.
– Custom module, state and proxy support. In /srv/salt, create a _modules directory, etc. They can either override the existing module, or add new ones.

Posted on

Ubuntu Unity application launchers for IntelliJ IDEA and PyCharm

Here’s how I got Ubuntu Unity to show application launchers for >IntelliJ IDEA and PyCharm

In $HOME/.local/share/applications, add an IDEA.desktop file with these contents:

#!/usr/bin/env xdg-open
[Desktop Entry]
Version=13
Name=IntelliJ IDEA
GenericName=Text Editor
Exec=/home/YourHomeDirectory/idea-IC-135.480/bin/idea.sh
Terminal=false
Icon=/home/YourHomeDirectory/idea-IC-135.480/bin/idea.png
Type=Application
Categories=TextEditor;IDE;Development
X-Ayatana-Desktop-Shortcuts=NewWindow
Icon[en_US]=/home/YourHomeDirectory/idea-IC-135.480/bin/idea.png

And a PyCharm.desktop file:

#!/usr/bin/env xdg-open
[Desktop Entry]
Version=3
Name=PyCharm
Exec=/home/YourHomeDirectory/pycharm-3.0.1/bin/pycharm.sh
Terminal=false
Icon=/home/YourHomeDirectory/pycharm-3.0.1/bin/pycharm.png
Type=Application
Categories=TextEditor;IDE;Development
X-Ayatana-Desktop-Shortcuts=NewWindow
Icon[en_US]=/home/YourHomeDirectory/pycharm-3.0.1/bin/pycharm.png

I use the default keyboard shortcuts, and on Linux, CTRL-ALT left arrow doesn’t work with PyCharm or IDEA (jumps back to where I was before I followed a symbol with CTRL-B). I’ve found that CTRL-WINDOWS-ALT left arrow does work. Same thing for many other shortcuts that use CTRL-ALT.

Posted on

Tools and trade-offs

Ten or more years ago, I purchased a Gerber multi-tool. It was less expensive than a Leatherman, and it was painted in my favorite color. Since then, I saw my brother’s Leatherman Wave, and wanted one. For Father’s Day this year, Janice gave me a Wave. I like it — it’s a sleek piece of engineering in every way. The screwdriver fits more of the screws that I encounter compared to the bulky phillips head on the Gerber. The saw works far better. It’s easier to access the Leatherman’s knives, saw and file.

Leatherman Wave vs Gerber multi-tools

However, I’ve found that when I’m out and about in the yard, the Gerber is easier to work with. It’s belt-holster is secured with velcro — easy not only to open, but to close. And opening the pliers is an easy one-handed flick-of-the wrist, which means I don’t have to put the tool in my left hand down in order to open the pliers like I do with the Leatherman.

The Gerber’s pliers can be used to pry things apart. The Leatherman folds up when I try the same thing. The Leatherman is a more dangerous tool to open and use — more likely to pinch or draw blood (as it has done on a few occasions).

The tools definitely have trade-offs. I admire the Leatherman most, but for regular use, the Gerber wins.

The same thing is true of operating systems. I’ve used Linux and Windows for years, and have wanted to try out Mac OS X and a MacBook Pro.

My wish came true. For the past two months, I’ve been using a MacBook Pro 17″ with Retina display. Not only is the engineering and design gorgeous, the screen is eye-poppingly clear and crisp. The battery lasts and lasts, and the laptop doesn’t get too warm. The track-pad is the best I’ve ever used, and I love the gesture support to go forward and back in the web browser, and the gestures to switch desktop workspaces, activate Mission Control and Launch Pad.

Yet the keyboard shortcuts have taken some getting used to compared to the ones I know from both Gnome (Linux) and Windows 7. I find that it’s easy to accidentally hit Command-Q when I meant to hit Command-W. Control-Left and Control-Right don’t work — it’s Option-Left and Option-Right to jump forward and backward by a word. Home and End go to the beginning and end of a document instead of the beginning and end of a line. There’s no built-in window-snapping like Windows 7 or Gnome-Shell.

When I switch back and forth between Mac, Linux, and Windows, I hit the wrong keyboard shortcuts. Is it Command-C or Control-C for copy? On-and-on it goes — sharp edges. But I’m getting used to context switching. I love the MacBook Pro.

Last week, a Gallego UltraPro arrived from System76. It’s a compact powerhouse, and best of all, the keyboard shortcuts work like I expect, and I can get work done faster because I’m used to the underlying OS and tools. When programs break, I know where to look and what utilities to use to solve the problems (I love having /proc available). But the UltraPro with Ubuntu has sharp edges as well. The screen is a bit small, and it’s not as beautiful as the Retina display. It doesn’t have a backlit keyboard, and so far, I haven’t been able to get Evolution to talk to our Exchange server (which isn’t a problem on Mac because it has Outlook).

I admire the refinement, the visual beauty of OS X and the design of the MacBook Pro. For regular use in getting my work done, the UltraPro and Ubuntu currently win.