Skip to main content ↑ ↓
Topic: do you REALIZE how much memory all the messages would take?!? (Read 8077 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

do you REALIZE how much memory all the messages would take?!?

I don't...
So I did some tests.
I changed a bit the code, not a real refactor, just a workaround to re-route the functions. O:-)
I measured the memory usage and the memory peak in 4 positions in the code:
just before the query ($messages_request)
 after the query (that in the modified code corresponds to after the loading of the messages in $context
in the template just before the beginning of the while loop
 in the template just after the end of the loop.
The I reloaded the page 3 times on two different topics: one with 100 messages, the other with 1062.

Obviously the last point is the one with the highest memory usage and peak, so for the moment I'll report only these.

100 messages:
current$contextdifference (%)
Usage:1076276011314378
+5.1
Peak:1090851711441978
+4.9
1062 messages:
current$contextdifference (%)
Usage:3804529844549458
+17.1
Peak:3884924044796746
+15.3
Bugs creator.
Features destroyer.
Template killer.

Re: do you REALIZE how much memory all the messages would take?!?

Reply #1 –

In your refactoring, did you pull every single message into $context before sending it to the template? That's really what the point of the callback is about, is that even with a mammoth topic, the peak memory usage should never be that high.

For comparison, consider the print page memory usage on a similar situation.

Re: do you REALIZE how much memory all the messages would take?!?

Reply #2 –

I did the same test with 2.1 and the results are comparable (except that Opera dies with the resulting page, while in Elk still works lol).

Well, it's not nice to see, but attached there is the Display.controller.php I used ( _debug is simply a function I use to print debug informations, basically a formatted version of print_r ;)).
txt because php is not within the allowed extensions :P

Re: do you REALIZE how much memory all the messages would take?!?

Reply #3 –

The first thing to note is that if you have it as a callback, you won't pull 44MB of memory use (most hosts allow 32MB) but I dread to think what will happen if you plug in 'All' posts into something like the Aeva thread under that situation (close to 7k posts IIRC)

Keeping the callback does allow you curtail peak memory usage, however I'd like to know how you're burning > 30MB when the peak should be only a modest fluctuation. I wonder if it's related to output buffering, actually.

Re: do you REALIZE how much memory all the messages would take?!?

Reply #4 –

Quote from: Arantor – The first thing to note is that if you have it as a callback, you won't pull 44MB of memory use (most hosts allow 32MB) but I dread to think what will happen if you plug in 'All' posts into something like the Aeva thread under that situation (close to 7k posts IIRC)
Well, the topic's is mostly because I really didn't know how much the callback is important (or not) and wanted to get figures.
For the 32MB limit it wouldn't change much because the callback uses 38, we'd be out anyway in the gigantic topic.
I forgot to take note of the time taken to create the page... ::)
The smaller topic is more similar to what I would expect from a classic forum and its memory usage (so about 10/11 MB).

Quote from: Arantor – Keeping the callback does allow you curtail peak memory usage, however I'd like to know how you're burning > 30MB when the peak should be only a modest fluctuation. I wonder if it's related to output buffering, actually.
Possible, output buffering is enabled, I didn't touch anything else apart from the code in Display.controller.php (and display.template.php.
Though...I don't have the data with me at the moment, but I seem to remember for the current method a trend similar to:
8 MB
 18 MB
28 MB
 38 MB
while for the one without the callback:
8 MB
 38 MB
40 MB
 44 MB
or something like that. Please do not elaborate too much on these numbers because I'm not so sure about them, this afternoon I'll post the real ones. ;)
Bugs creator.
Features destroyer.
Template killer.

Re: do you REALIZE how much memory all the messages would take?!?

Reply #5 –

Here they are.

Memory usage:
Current$contextdifference (%)
18
18
0.1
18
40
123
18
40
122
38
45
17.1
Memory peak:
Current$contextdifference (%)
19
19
0.1
19
40
117
19
41
118
39
45
15.3
Bugs creator.
Features destroyer.
Template killer.

Re: do you REALIZE how much memory all the messages would take?!?

Reply #6 –

Hmm, the unbuffered aspect makes sense. SELECT * on the other hand... not so much. Yes, pulling every column from a table is so much more efficient than only pulling the ones you need (though for messages, it largely is every column)

SMF and Elk as they stand cannot use unbuffered queries because of the DB abstraction (PGSQL doesn't support it, don't believe SQLite does either) but maybe Wedge could.

It's just weird that you're getting those figures, because cases of large amounts of messages can occur (all posts in a topic at once)... makes me wonder how much of that memory usage is tied to attachments and members and not the posts as a whole.

Re: do you REALIZE how much memory all the messages would take?!?

Reply #7 –

Attachments none because is a randomly generated forum created using populate and it doesn't add attachments.

Members...mmm...possible since in the topic there are 1054 different members (result of count($posters)).
I'll generate a similar topic with just 1 member to see the difference.
Bugs creator.
Features destroyer.
Template killer.

Re: do you REALIZE how much memory all the messages would take?!?

Reply #8 –

Tell you what I'd do, actually... set $context = array() just before pulling everything, taking the memory at that time (not peak) and taking the memory after that... if memory use is already 10MB on 100 messages, and 40MB on 1000 messages, 900 messages = 30MB. So how much of that 10MB is other overhead?

Re: do you REALIZE how much memory all the messages would take?!?

Reply #9 –

Sorry, but with that attitude you just lost the last bit of credibility I had for you as a developer, which from your previous posting wasn't exactly high to start with (like quoting the buzzword of the week, e.g. recently talking about building a forum on MongoDB... answer: unless you're denormalising everything for mass duplication of data, don't go there, MongoDB isn't relational)

There are plenty of good reasons you DON'T do that, especially in MySQL., it's an excellent way to get you into filesorts and table scans far more often because you're pulling in columns you probably don't need - and you can't alias columns particularly easily. Unless you explicitly need every column, you really shouldn't. And even if you do need every column now you might not need it later.

http://stackoverflow.com/questions/65512/which-is-faster-best-select-or-select-column1-colum2-column3-etc for example gives plenty of reasons why you shouldn't. Though the one argument about column ordering is of course not relevant if you're using fetch_assoc and similar, but that's small potatoes.

In the case even being talked about, this is why I bring it up - the amount of memory reserved by member data is actually quite surprising and may take up a larger percentage of the memory use being talked about here. It's not all needed but it's all pulled.

Re: do you REALIZE how much memory all the messages would take?!?

Reply #10 –

In typical Arantor fashion, the topic is being pulled to superfluous debate and name-calling. I'll withdraw everything I posted and go on my merry way.

Re: do you REALIZE how much memory all the messages would take?!?

Reply #11 –

Holy crap, for a second I thought I was on the SMF site  :'(

Missed part of the debate, sounds like some of it was about using SELECT * which I don't like and had commented to that end in one of the PR's.  The thread pointed to by Arantor is a good read, thanks for that link.

Back on the memory usage, interesting test to run, I'm not sure what I would have expected to see either, so can't say I'm surprised by the results one way of the other (in terms of the scale of the savings).   Maybe php4 had a bigger gain, you can always run the test on SMF 2.0  and see O:-)