Author: Jared

Posted on

jq: sed/grep-like tool for processing JSON data

The other day I was exporting json data from mongodb using mongoexport, and needed a way to exclude certain subdocument fields. mongoexport doesn’t support excluding fields.

jq saved the day.

I piped the output of mongoexport through jq, and piped the output of that to gzip. My pipeline looks like this:

nice mongoexport –db mydb –collection accounts –out – | nice jq -c -M ‘del(.phash) | del(.h[].phash)’ | nice gzip > accounts.json.gz

Whoever wrote jq, thank you.

Posted on

Introduction to SaltStack

I had the opportunity to give an introduction to SaltStack at the OpenWest conference this week (Video on YouTube).

As I was setting up for the presentation, I realized I needed an HDMI cable, and there wasn’t one in the room. A conference helper found a cable at the last minute, and we started on time (thank you!).

From what the audience said, many people are interested in using SaltStack, but haven’t started yet. It’s a fantastic tool for remote execution and configuration management. The documentation is initially a bit intimidating, but it’s easier to get started than it appears — especially using one of he many online tutorials. And there are many examples available from saltstarters.org.

SaltStack releases for CentOS/RHEL have become more stable over the past 10 months, and I’m excited about it’s future. It has a vibrant development community, and they’re implementing cool new features to extend its capabilities, including:

– RAET, an optional new, secure UDP communications protocol, which will easily support more than 10,000 minions. https://github.com/saltstack/raet
– Proxy minions will eventually allow the configuration of load balancers and networking equipment.

And advanced features, already present, sound like they could be useful:

– The orchestration framework, based around the reactor, can allow configuration when certain events are triggered. For example, Jenkins, upon building successfully, could send an event from the Salt minion to the master, and the Salt Master could automatically deploy the build to a test environment.
– Custom module, state and proxy support. In /srv/salt, create a _modules directory, etc. They can either override the existing module, or add new ones.

Posted on

Ubuntu Unity application launchers for IntelliJ IDEA and PyCharm

Here’s how I got Ubuntu Unity to show application launchers for >IntelliJ IDEA and PyCharm

In $HOME/.local/share/applications, add an IDEA.desktop file with these contents:

#!/usr/bin/env xdg-open
[Desktop Entry]
Version=13
Name=IntelliJ IDEA
GenericName=Text Editor
Exec=/home/YourHomeDirectory/idea-IC-135.480/bin/idea.sh
Terminal=false
Icon=/home/YourHomeDirectory/idea-IC-135.480/bin/idea.png
Type=Application
Categories=TextEditor;IDE;Development
X-Ayatana-Desktop-Shortcuts=NewWindow
Icon[en_US]=/home/YourHomeDirectory/idea-IC-135.480/bin/idea.png

And a PyCharm.desktop file:

#!/usr/bin/env xdg-open
[Desktop Entry]
Version=3
Name=PyCharm
Exec=/home/YourHomeDirectory/pycharm-3.0.1/bin/pycharm.sh
Terminal=false
Icon=/home/YourHomeDirectory/pycharm-3.0.1/bin/pycharm.png
Type=Application
Categories=TextEditor;IDE;Development
X-Ayatana-Desktop-Shortcuts=NewWindow
Icon[en_US]=/home/YourHomeDirectory/pycharm-3.0.1/bin/pycharm.png

I use the default keyboard shortcuts, and on Linux, CTRL-ALT left arrow doesn’t work with PyCharm or IDEA (jumps back to where I was before I followed a symbol with CTRL-B). I’ve found that CTRL-WINDOWS-ALT left arrow does work. Same thing for many other shortcuts that use CTRL-ALT.

Posted on

Galago UltraPro laptop: the good and the bad

I’ve had a Galago UltraPro laptop from System76 since August of 2013, and I use it every day at work (thank you, Vivint). Overall, I love it — but would have looked for another option had I known about the bad parts.

The good:

  1. Ships with Ubuntu by default, including drivers for the hardware.
  2. Powerful: Intel Haswell processor with the Intel Iris Pro graphics chip, which means it’s fast, and it can drive a Dell 30″ monitor (using an Apple mini display port to dual-link DVI adapter), an HDMI monitor, and the laptop screen at the same time.
  3. Small and lightweight — easy to carry around.
  4. USB 3.0
  5. The keyboard layout and function keys are designed for Linux, and they work — no tweaking necessary.
  6. The touch pad works well (although it’s not as awesome as Apple’s touchpad and gestures)

The bad:

  1. The screen is too small to use at its high resolution unless it’s sitting on my lap. So I always use external monitors when I’m sitting at my desk.
  2. No backlit keyboard.
  3. No indicators/lights for the caps lock, scroll lock and num lock keys — so you don’t know what state your keyboard is in. I hate this — it’s a huge omission.
  4. The ethernet jack door flips down, and it breaks off easily. When that happens, the ethernet cable doesn’t stay plugged in very easily. Inexcusable. It’s possible to use a USB-to-Ethernet device, but who wants to do that?

The Galago UltraPro is a fantastic Linux workstation, but it’s a poor laptop compared to most other laptops (with the exception of being lightweight) because the screen is too small, the keyboard isn’t backlit, and the lock keys lack indicator lights. A MacBook Pro Retina is a better laptop in almost every way. The screen is oh-so-beautiful, the keys are backlit, and the caps lock key tells you when it’s on (but the function key is in the wrong place — the control key should go there — what was Apple thinking?).

Why can’t Apple ship Ubuntu as an option on the MacBook Pro? It would be awesome, because they’d support the hardware with Linux drivers.

I can dream.

Posted on

Article: Facebook tracks self-censorship and more

Ars writer Casey Johnston reports:

Facebook released a study (PDF) last week indicating that the company is moving into a new type of data collection in earnest: the things we do not say

Facebook’s users are its product — without the content we post and the interactions we have, Facebook wouldn’t be able to sell ads, gaming, data sets, etc. It’s not just Facebook — it’s also true of Google, Microsoft, Yahoo and Apple. Every large company has computers analyzing what we do and don’t do, using that information to its economic advantage.

Posted on

HDMI audio output and the power of a community

Tonight, I wanted to show a KSL.com news story to my children, and when I plugged the HDMI cable into our Windows 8.1 laptop, we saw video, but there was no accompanying audio. 🙁

Fortunately, a google search for “Windows 8 HDMI audio” yielded a solution from ‘wasala18’, who posted a link to an Intel driver:

https://downloadcenter.intel.com/Detail_Desc.aspx?agr=Y&DwnldID=23323&lang=eng

that link takes u to an update from Intel for they’re hd graphics and audio drivers.. I found the solution for my laptop NO THANKS TO HP OR MICROSOFT!! and these updated drivers aren’t even available through windows update.. just plain sad.. I still stand by everything i said before..

I’m grateful for the power of community to help solve problems, for Google searches, and for Intel’s drivers.

Now we can enjoy HDMI audio piped through the stereo.

Posted on

vCloud Director CentOS Guest OS Customization Script

I needed to customize a custom Linux VM image with a static IP address in vCloud Director. Here’s the script I came up with. It doesn’t work unless VMWare tools is installed and running in the VM (kernel upgrades tend to break it). The VM should be stopped, then paste the script into the “Guest OS Customization” script area. Boot the VM with the “Power On and Force Recustomization” option.

 

#!/bin/bash
HOSTNAME=YourHostNameHere
IPADDR=StaticIpAddressGoesHere
NETMASK=255.255.255.0
(
echo "------------------------------------------------------------------------------"
date
if [ x$1 == x"precustomization" ]; then
  set -e
  echo "Running Pre-Customization tasks..."
  if [ x$HOSTNAME != x ]; then
    echo "Configuring hostname as $HOSTNAME"
    sed -i "/HOSTNAME/d" /etc/sysconfig/network
    echo "HOSTNAME=$HOSTNAME" >> /etc/sysconfig/network 
    hostname $HOSTNAME
    hostname
  else
    # Set it so it can be used below
    HOSTNAME=`hostname`
  fi
  if [ x$IPADDR != x ]; then
    echo "Configuring network IP=$IPADDR NETMASK=$NETMASK"
    rm /etc/udev/rule.d/70-persistent-net.rules
    sed -i "/IPADDR/d" /etc/sysconfig/network-scripts/ifcfg-eth0
    sed -i "/NETMASK/d" /etc/sysconfig/network-scripts/ifcfg-eth0
    sed -i "/HWADDR/d" /etc/sysconfig/network-scripts/ifcfg-eth0
    sed -i "/BOOTPROTO/d" /etc/sysconfig/network-scripts/ifcfg-eth0
    sed -i "/DHCPHOSTNAME/d" /etc/sysconfig/network-scripts/ifcfg-eth0
    cat >> /etc/sysconfig/network-scripts/ifcfg-eth0 <<ENDCFG
BOOTPROTO=static
IPADDR=$IPADDR
NETMASK=$NETMASK
DHCPHOSTNAME=$HOSTNAME
ENDCFG
  fi
  echo "Done with Pre-Customization"
elif [ x$1 == x"postcustomization" ]; then
  echo "Running Post-Customization tasks..."
fi
) | tee -a /root/customize.log

 

Update: It turns out that having VMWare Tools functioning properly is enough for vCloud Director to configure the network settings all by itself, without the help of a script.