Recently, I was asked to constrain the memory usage of an application on Linux. Glibc provides hooks for malloc, free, etc. By the way, the hook functions are responsible to guarantee thread safety -- glibc doesn't do it automatically. I used the malloc hooks in combination with a memory manager that a colleague found: TLSF. There are two implementations:

• http://tlsf.baisoku.org/ (public domain)
• http://rtportal.upv.es/rtmalloc/ (GPL)

There are benefits and caveats when using a custom memory allocator. TLSF was meant to shine for real-time use, because the overhead of malloc and free are O(1) constant-time operations. On the other hand, TLSF isn't thread-safe.