Posts

Posted on

RabbitMQ, memcache, and too many socket connections

What happens when you have hundreds of services connected to RabbitMQ and memcache, and those services have a bug that causes them to keep their previous socket connections open, and repeatedly reconnect to RabbitMQ and memcache?

They crash.

It occurred to me that one can prevent too many connections using iptables on the RabbitMQ and memcache machines. Here’s how:

http://www.cyberciti.biz/faq/iptables-connection-limits-howto/

The corollary is that setting the per-ip connection limit too low can also cause problems.

I’d guess that more commonly public-facing servers like NGINX and Apache don’t have the problem of crashing. Hopefully, they degrade gracefully, and refuse additional connections while continuing to service the connections they already have open.

Posted on

Great tools: ag and rlwrap

It’s fun to learn about new command line tools from coworkers. Here are two.

  1. rlwrap can be used to wrap anything in a realine command history. It’s useful to preserve command history, including the commands typed in remote ssh sessions. Just wrap ssh in rlwrap.
  2. ag, the silver searcher, is a super-fast recursive grep tool. I enjoy using it, and am pleased at how quickly it returns search results on my source code trees.
Posted on

LWN.net: “Changes in the TLS certificate ecosystem”

I was glad to come up to speed with what has been happening with TLS in the last couple of years, and I highly recommend reading these articles.

  • https://lwn.net/Articles/663875/
  • https://lwn.net/Articles/664385/

I learned about HTTP Public Key Pinning, Certificate Transparency, and STARTTLS stripping, among other things.

Here’s one of many good quotes:

The core problem of the TLS certificate system is that there exist hundreds of certificate authorities. And unless extra protection measures are in place, each of those can create valid certificates for any domain. Therefore the whole system is only as strong as the weakest of all certificate authorities.

And as for embedded devices that handle encryption:

We are well aware that crypto appears to be something that needs to be field replaceable, and yet we more or less have no clue how to do that in deployed embedded hardware. Indeed, we seem to have a very poor idea in general on how to maintain the software on field deployed embedded hardware. — Perry Metzger

As I’ve worked with Python, I realize that it’s one thing to implement TLS, and another thing to verify server certificates. The Python requests library can be configured to do the right thing, but the python SMTP cannot. It’s still another thing to check on certificate revocation. Python doesn’t implement OCSP or CRLs, and those mechanisms are problematic anyway. It doesn’t yet implement HTTP Public Key Pinning. The state of affairs may not be much better in other programming toolboxes.

So I’d guess that machine to machine internet communication is probably more vulnerable to man in the middle attacks than consumer web browsers.

Posted on

Unsatisfactory Freedompop cellular experience

In September, my son started junior high, and he craved having a smartphone. His lawn-mowing money was burning a hole in his self-made duct-tape wallet. So I googled for inexpensive options. Freedompop sounded like a great deal — free phone service (500 MB data per month), based on VoIP over cellular data. Too good to be true? Read on.

We ordered a Samsung Galaxy S4 that was supposed to take up to three weeks to be delivered. The website listed the phone as sitting in a “kitting” stage for the duration, and still no phone, and no follow-up email to tell us whether our order had disappeared into a black hole.

I couldn’t find a phone number for tech support, and there was no “live chat” option on the freedompop website, so I turned to Google, and they came to the rescue. I called, navigated a deep option tree, and eventually reached a live person, which was no easy feat. They told me the phones were on back order, and to wait another week or two.

So I did, with a very anxious son asking me daily about the order status. I showed him how to check the status himself. When two weeks had passed, I called again, and for the second time, I was told the phone was on back order, but it should be coming in the next week or two. Two weeks later, the phone finally shipped. My son was so excited to open the box and play with his shiny new device.

Knowing that smart phones love to consume data, we disabled most of the apps from using cellular data.

In the coming two weeks, I received emails from freedompop saying that cellular data had run out, and that I had been charged $10 to “top-up”. That was a surprise, so I disabled that feature, using their website.

I opened the cellular data usage graph on the smartphone. It told me that the app known as “Android OS” had helpfully been chewing up the cellular data in large stair-step usage patterns. Android doesn’t allow us to disable cellular data for “Android OS”.

Screenshot_2015-11-04-21-46-08 (1)

Apparently the high cellular data usage correlates with the downloaded of a 263 MB OS update over cellular. I recalled that we tried installing the update a few times, and it didn’t work because FreedomPop modifies the phone so that OS updates don’t install.

So the phone is caught in an impossible cycle, and the FreedomPop experience has been anything but a good one.

I submitted an email to their support system, which told me I would have a response within 48 to 72 hours. It’s been five days, and I’ve heard nothing further. I tried calling tech support, and after waiting on the line for 25 minutes, had to hang up.

What I thought we would save in cellular costs, I’ve paid for in time and top-up charges. The promise of “free” was too good to be true.

I wish we had chosen Ting, which has phenomenal customer service.

Posted on

MongoDB: Pre-splitting a sharded collection

When suddenly writing high volumes of data to a MongoDB collection that’s had little or no data previously, it’s important to pre-split the collection so that there’s good write performance — we don’t want to write all data to a single shard while waiting for the MongoDB balancer to figure things out. While it’s possible to programattically specify the split points in advance, MongoDB has an easier way: Hashed shard keys. E.g.:

db.adminCommand({shardCollection: 'test.user', \
  key: {uid: 'hashed'}, \
  numInitialChunks: 500})

Equivalent python code looks approximately like this:

my_pymongo_connection["admin"].command("shardcollection", "test.user", \
  key={'uid': 'hashed'}, \
  numInitialChunks=500)

The downside is that it can take a while to shard the collection — the call doesn’t return until it’s complete, and it reportedly blocks all other clients of a given mongos instance until it finishes.

Apparently it’s not possible to specify more than 8129 initial chunks.

Posted on

Data security can only be achieved by those empowered

Users of online services don’t have the ability (i.e. aren’t empowered) to secure the data stored by those services. Only the engineers and the companies that build the services can do that. So I agree with Cindy Cohn, who says:

…we need to ensure that companies to whom we entrust our data have clear, enforceable obligations to keep it safe from bad guys. This includes those who handle it it directly and those who build the tools we use to store or otherwise handle it ourselves.

In my view, business leadership and software engineers have an ethical responsibility to secure their systems and services so that customer’s data and sensitive information doesn’t get misused or abused.

I’d like it if customers had a reliable and consistent way to evaluate the quality and diligence given to keeping their data safe — something like Charity watch or Consumer Reports.

Posted on

Continuous Delivery

Have you been working on a software project where your momentum seems to be slowing down? It tends to happen as features are added, because it requires time and effort to maintain and verify existing features with each and every release. Without automated tests, momentum slows, or worse, you end up shipping broken software from time to time.

A book titled Continuous Delivery enumerates changes and improvements that organizations can adopt to increase momentum. The business case seems compelling, with the promise of:

  1. Faster reaction times (for the business, for new features, for bugfixes)
  2. Reduced risk via earlier feedback
  3. Flexibility in releasing (dormant features, enabled with a feature switch)
  4. Reduced development costs

The authors have an informative website, and others, including Atlassian, have written on the topic of Continuous Delivery in an informative series of blog posts called “A skeptic’s guide to continuous delivery”, split into parts one, two, three, four and five. In the first installment, they say “CD fundamentally requires some amount of cultural rewiring because the core structure of CD is a delivery pipeline through which changes flow”.

I like the vision of Continuous Delivery, and if large and small companies have used it to increase their agility in achieving business objectives, then it’s possible for others to do the same.

Posted on

pre-commit

At work, we use git, and git supports hooks, including pre-commit hooks. Rather than write my own, and do it poorly, I’m using a tool called pre-commit, created by engineers at Yelp.com. To them, I offer my thanks.