12:31 pm, 14 Dec 06
tcmalloc
Wikipedia using tcmalloc:
[...] Quite a few problems resulted in memory leaks, but one was particularly nasty: Squid processes under high load started leaking CPU cycles somewhere. After deploying profiling for squid we actually ended up seeing that the problem is inside libc. Once we started profiling libc, one of initial assumptions appeared to be true - our heap was awfully fragmented, slowing down malloc().The tcmalloc documentation is pretty detailed in describing how it works, too.
Here comes our steroids part: Google has developed a drop-in malloc replacement, tcmalloc, that is really efficient. Space efficient, cpu efficient, lock efficient. This is probably mostly used (and sophisticated) libc function, that was suffering performance issues not that many people wanted to actually tackle. The description sounded really nice, so we ended up using it for our suffering Squids.
The results were what we expected - awesome :)
I've been tracking it for over a year now for Fedora, and the response from upstream has been extremely disappointing.
http://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=170303
I don't mean to imply that we have received no response in the last year+, but it does illustrate our ongoing frustration.
(And conversely, one of the reasons I ♥ Lua, despite some issues with its syntax.)
6MB is unacceptable for many applications, but of course it's a lot less than it used to be.
you really find this impressive?
> it can easily support many of the features that aren't built-in
is it Turing-complete, too?
(don't take my tone personally, please. I happen to believe people are too quick to design "new" programming languages, and I think this is not healthy at all. hell, just look at the almost invariably weep-inducing post-hoc rationalizations!).
anyways, the point of the comment above was that you can fit GC/closures/REPL in much smaller space than 1M, so I don't see how the cited feat is, in itself, impressive.
FWIW, I haven't looked at Lua in detail but I do know it is the one of the most popular languages for embedding and chosen by a variety of independent projects, which confirms to me that there must be something technically appealing about it.
sorry! pet peeves etc.
> I do know it is the one of the most popular languages for embedding
indeed. I've actually looked at Lua a while back, because it is (was at the time, anyway) the "extension" language of the Elinks web browser.
it caught on, basically, because:
1. it had sane lexical scoping, closures and dynamic typing while looking neither like Lisp nor like Perl.
2. its (only) implementation was light and designed to be embedded.
it is certainly an engineering and marketing success.
Jesus, go take five minutes to read the docs on the website if you're so curious. The stuff about the register-based VM or coroutine support might interest you. Oh wait, those were rhetorical questions, weren't they. You actually don't care about the language, you're just trying to get across a broad generalization that New Languages Are Bad.
I'm not taking your tone personally, but it's nonetheless really annoying. I didn't ask for an argument, I was just pointing out something that I find interesting and useful. This is the point where I put on a fixed distant smile and start backing slowly away.
And of course don't take any of what I just said personally.
New Languages designed for cheap short-lived success by taking a set of proven ideas and gratuitiously tweaking the syntax Are, indeed, Bad. in the background I'm also noting the regrettable (if extremely widespread) language/implementation confusion you seem to exhibit.
> I'm not taking your tone personally, but it's nonetheless really annoying.
sorry! (really. when I start conversing with people, I hope to walk out of the conversation having learned something new and interesting).
GC and closures are not magic; they are not even a big deal, size-wise.
most of those 5.5MB footprints you are seeing is "library" (ANSI CL is a featureful language, Perl even more so).
(as for the Schemes I have on this box: Tinyscheme takes 768K, SCM would probably be similar or even less if it didn't insist on loading most of SLIB before even displaying the prompt; etc. I wonder how much memory LispMe would take if it were targeted at x86 :)).
There isn't really a CPAN/PEAR/Gems equivalent yet, although there is a LuaForge site.
MyClass = { new = function () { object = ... ... }; meth1 = function () { ... meth2 ... }; meth2 = ... };Till this day it gives me the creeps that in C++ I have to define stuff with MyClass::MyMeth all the time.(Of course I realize that this is a syntax thing, but as syntactical principles go I think declarative style is one of the deeper. Specifically in regard to namespaces, their whole purpose is an organizational aid; if the language doesn't provide them natively much of their appeal is lost. I can do
myapp_myclass_new()in c too...)So tcmalloc is (obviously) no silver bullet, and it seems probable that the libc malloc was chosen as a balance between all sorts of competing goals of size and performance and debuggability. I mostly think this is interesting because it shows another design point that is, for many applications, a quite reasonable trade-off, and one I hadn't really thought about before.
I also thought their point about applications often resorting to their own memory management interesting. Even desktop-oriented libraries do this.
I've had occasion to use GMemChunk, the older, now obsoleted, API. I looked at the code to make sure that it will work well, too. IIRC the reason for its replacement by the slices stuff was expensive or unpredictable housekeeping, which I worked around in the typical cop-out way of allocating more memory than I strictly needed. It fit my application and performed just fine in high-moose production.
otherwise, you may hit cache problems due to its "let's group objects together by size!" design decision. I know a certain product that has rejected the Boehm allocator for this very reason.
then again, heavily-threaded C/C++ programs probably don't have to worry about cache performance due to missing the cache all the bloody time anyway. :)
Re: Reply to your comment...
if you have a critical loop that is carefully coded to access a carefully-chosen and similarly carefully-allocated set of things in memory, then the less that loop has its context swapped in and out the better it will perform.Re: Reply to your comment...
why, thank you!(I'll be the first to admit that where it comes to threads, I may well be talking out of my ass. I fear threads).
Re: Reply to your comment...
And the less it can do!