There’s a book titled “Measure What Matters”, and it’s fascinating and worth the read. This article from acmqueue with nearly the same title is quite interesting, because it turns some of my previous thinking on it’s head.
Summary: “Speed matters”; Instrument client code to measure user experience; Measure “long-tail” latency at the 95th and 99th percentiles; even minor changes to code or user behavior can effect things; build bench-marking into release testing procedures.