My site on Zettai has been very stable for a a few weeks, but, in the 
last couple of days, something seems to have gone wrong and I've had to 
restart Zope a few times.  Also, just a little while ago, I tried to 
pack the ZODB and got a Site Error - Memory Error.  So, I restarted 
again.

Brief snapshot of my environment:
* plone 2.0.5
* Zope 2.7.4-0, python 2.3.4
* my max memory is set at 100MB at the Zettai account.  They are 
supposed to running a memory check monitor - not sure why it is not 
kicking in.

I have these questions:

* I did something that was stupid a couple days ago and I'm wondering 
if this may be causing some instability in the ZODB or causing a memory 
leak.  What I did was create a new topic and copy/paste all the items 
from an existing topic.  The topic was a list of site comments.  This 
took  a very long time to run and then it dawned on me that of course 
this was not a good thing to do, since a topic is a virtual folder - 
i'm guessing with pointers to the actual objects.  So, I deleted this 
new topic and created a new one the correct way by specifying criteria. 
   But, I'm wondering if my original mistake could have caused the 
problems I'm experiencing?  And, if so, what can I do about it?

* Does anyone have any suggestions to diagnose what's going on in my 
site, or other ideas?   I was hoping to release the site for production 
at end of the coming week and so these problems could not have come at 
a worse time for me.

thanks very much.

David

P.S. I sent a similar message to plone-users and have not received any 
replies.  Is there something wrong with the way I stated it or is it 
too vague?

==============================
David Diskin, david.diskin(at)verizon.net

On Mon, May 16, 2005 at 11:05:33PM -0400, David Diskin wrote:[...]

I'm not quite sure what you did with your topic, but it's highly
unlikely to be related.

You mentioned getting a MemoryError while packing, and that you
had to restart Zope a few times, but you haven't mentioned *why*
you had to restart Zope--what was wrong? Was it hanging? Were you
getting errors? If so, what?

Have you looked in the events log for anything applicable?

- j.

hi Joel -

Zope was "frozen" and "hanging" - I could not access the plone site or 
the zmi at Zettai.  I got to the server with ssh, and zope was at the 
max memory allowed by Zettai for my account - 100 MB.  So, I had to 
restart and then the site was operational again.

The event log is big.  I've attached it - perhaps you can make some 
sense of it?

Thanks so much!

David

Zettai has not been terribly helpful

Joel -

Here's a list of the installed products.  I wonder if one of them could 
be causing memory leak??

David

David - I experienced some Memory Errors before with Zope, and it was 
due to the limits imposed for the user (limit, ulimit, or something like 
that). I was uploading a bunch of pictures via FTP, and when Zope 
reached a certain amount of memory (in my case 256MB), Zope stop working 
due to a Memory Error. In your case, instead of the limits it is 
probably the 100MB limit at the Zettai account, which it could be a 
product leak, but unlikely if you did not install a new product in the 
last week.

The logs shows a couple of MemoryError distributed among the following 
times:

(FIRST)
2005-05-16  07:06:14

(SECOND)
2005-05-16T11:24:29 INFO(0) Zope Ready to handle requests
2005-05-16T17:31:34
2005-05-16T19:47:18

I wonder if the Memory Error were caused after you tried to copy/paste 
the topics, which may exhaust the memory allocation.

Also, I wonder what is the memory consumption after you restart Zope?

Just some thoughts,


David Diskin wrote:
[...]

Jordi -

The memory consumption after restarting is around 60MB.  I think you're  
right about  the memory allocation being exhausted after the copy/paste  
of topics.  Is 100 MB way too low for a memory limit?  Thanks.

David
On May 17, 2005, at 10:47 AM, Jordi Yeh wrote:
[...][...][...]
==============================
David Diskin, david.diskin(at)verizon.net

The very fastest way to "fix" your memory limit problem is to reduce  
the number of threads you have in Zope.  You are probably running with  
four threads by default.  You can set the zserver-threads configuration  
parameter in etc/zope.conf to lower this number.  I'd suggest going to  
2, and seeing how that does for you.

You can fiddle with tuning the object cache size, but generally those  
numbers get tuned upwards for performance, not downwards for cost.


On May 17, 2005, at 11:16 AM, David Diskin wrote:
[...][...]
>>> the last couple of days, something seems to have gone wrong and
I've  
>>> had to restart Zope a few times. Also, just a little while ago, I 

>>> tried to pack the ZODB and got a Site Error - Memory Error. So, I 

>>> restarted again.
>>>
>>> Brief snapshot of my environment:
>>> * plone 2.0.5
>>> * Zope 2.7.4-0, python 2.3.4
>>> * my max memory is set at 100MB at the Zettai account. They are  
>>> supposed to running a memory check monitor - not sure why it is
not  
>>> kicking in.
>>>
>>> I have these questions:
>>>
>>> * I did something that was stupid a couple days ago and I'm  
>>> wondering if this may be causing some instability in the ZODB or  
>>> causing a memory leak. What I did was create a new topic and  
>>> copy/paste all the items from an existing topic. The topic was a  
>>> list of site comments. This took a very long time to run and then
it  
>>> dawned on me that of course this was not a good thing to do, since
a  
>>> topic is a virtual folder - i'm guessing with pointers to the
actual  
>>> objects. So, I deleted this new topic and created a new one the  
>>> correct way by specifying criteria. But, I'm wondering if my  
>>> original mistake could have caused the problems I'm experiencing? 

>>> And, if so, what can I do about it?
>>>
>>> * Does anyone have any suggestions to diagnose what's going on in
my  
>>> site, or other ideas? I was hoping to release the site for  
>>> production at end of the coming week and so these problems could
not  
>>> have come at a worse time for me.
>>>
>>> thanks very much.
>>>
>>> David
>>>[...][...]

Thanks, Matthew.  I adjusted the threads to 3 for now.  It has helped.

David

On May 17, 2005, at 11:20 AM, Matthew T.Kromer wrote:
[...][...]
>>> was due to the limits imposed for the user (limit, ulimit, or  
>>> something like that). I was uploading a bunch of pictures via FTP,
 
>>> and when Zope reached a certain amount of memory (in my case
256MB),  
>>> Zope stop working due to a Memory Error. In your case, instead of 

>>> the limits it is probably the 100MB limit at the Zettai account,  
>>> which it could be a product leak, but unlikely if you did not  
>>> install a new product in the last week.
>>>
>>> The logs shows a couple of MemoryError distributed among the  
>>> following times:
>>>
>>> (FIRST)
>>> 2005-05-16  07:06:14
>>>
>>> (SECOND)
>>> 2005-05-16T11:24:29 INFO(0) Zope Ready to handle requests
>>> 2005-05-16T17:31:34
>>> 2005-05-16T19:47:18
>>>
>>> I wonder if the Memory Error were caused after you tried to  
>>> copy/paste the topics, which may exhaust the memory allocation.
>>>
>>> Also, I wonder what is the memory consumption after you restart
Zope?
>>>
>>> Just some thoughts,
>>>
>>>
>>> David Diskin wrote:
>>>
>>>> My site on Zettai has been very stable for a a few weeks, but,
in  
>>>> the last couple of days, something seems to have gone wrong
and  
>>>> I've had to restart Zope a few times. Also, just a little
while  
>>>> ago, I tried to pack the ZODB and got a Site Error - Memory
Error.  
>>>> So, I restarted again.
>>>>
>>>> Brief snapshot of my environment:
>>>> * plone 2.0.5
>>>> * Zope 2.7.4-0, python 2.3.4
>>>> * my max memory is set at 100MB at the Zettai account. They
are  
>>>> supposed to running a memory check monitor - not sure why it
is not  
>>>> kicking in.
>>>>
>>>> I have these questions:
>>>>
>>>> * I did something that was stupid a couple days ago and I'm  
>>>> wondering if this may be causing some instability in the ZODB
or  
>>>> causing a memory leak. What I did was create a new topic and  
>>>> copy/paste all the items from an existing topic. The topic was
a  
>>>> list of site comments. This took a very long time to run and
then  
>>>> it dawned on me that of course this was not a good thing to
do,  
>>>> since a topic is a virtual folder - i'm guessing with pointers
to  
>>>> the actual objects. So, I deleted this new topic and created a
new  
>>>> one the correct way by specifying criteria. But, I'm wondering
if  
>>>> my original mistake could have caused the problems I'm  
>>>> experiencing? And, if so, what can I do about it?
>>>>
>>>> * Does anyone have any suggestions to diagnose what's going on
in  
>>>> my site, or other ideas? I was hoping to release the site for 

>>>> production at end of the coming week and so these problems
could  
>>>> not have come at a worse time for me.
>>>>
>>>> thanks very much.
>>>>
>>>> David
>>>>
>>>
>>>
>>> -- To unsubscribe send an email with subject 'unsubscribe' to  
>>> zpugdc(at)lists.zpugdc.org.
>>> Please contact zpugdc-owner(at)lists.zpugdc.org for questions.
>>> http://zpugdc.org/lists/zpugdc/archive/2005/2005-05/1116299136098/

>>> 1116341300098
>>>
>>>[...][...]
==============================
David Diskin, david.diskin(at)verizon.net

Its probably worth pointing out that in my experience, its rare to see 
Zope actually be able to use more than about 1.4 threads (yes, its a 
special feature to have .4 of a thread -- make sure you have all 
patches applied).

The reason for this is the Python GIL and how it interacts with your 
operating system's scheduler.   Windows may in fact do better than 
Linux when it comes to guaranteeing timeslices to application threads.  
What happens is that Python holds a single lock for control of the 
interpreter.  Only the thread that has this lock can execute Python 
calls or bytecodes.  The main loop in python counts the number of 
bytecodes executed, and after a certain number it will yield the lock 
and attempt to re-acquire it.  On linux, the same process that yielded 
it is usually the one to re-acquire it, thus 'starving out' any other 
threads.  As such, the only time other threads can execute is when the 
"main" (or at least, favored) thread is blocking on I/O.

In Zope, each application thread has its own private ZODB connection, 
so if two different threads are executing, they'll both load private 
copies from the ZODB.  The ZODB cache limits are enforced between 
requests, so its possible for an executing thread to consume a LOT of 
memory.  While you may not be able to tune the application down much, 
you can at least prevent extra copies of objects from being cached on 
threads that are doomed to starve by the way the OS handles lock 
contention.

When I've done performance tests in the past, I've found it to be 
extremely rare to get much use out of having multiple application 
threads in Zope.    Having more threads *can* increase throughput, but 
at the cost of increased latency.  The only reason I recommend having 
more than one is so that you can have a management thread around to 
observe a long-running request.  (Usually this doesn't help, as most 
users will simply click "refresh" on whatever it was that sent Zope 
into a tailspin in the first place, exacerbating the problem).

It's also interesting to point out that you can very easily make things 
WORSE by running Zope on a machine with multiple CPUs.  There's still 
only one GIL, except now there's a second CPU trying to get it.   The 
pattern for the unlucky CPU(s) goes like:  1) Try to get lock.   2) 
Fail, block on lock.  3) Get signal from OS that lock has been 
released.  4)  Go back to step 1.   At least with a single CPU system, 
the CPU that is releasing a lock can't be simultaneously trying to 
acquire it.   Thrashing between CPUs can add more latency to the 
system.

To see this as an example, consider two processes, A and B trying to 
count to 10 with the scheduling granularity set so fine that they 
alternate every tick.

1-A(1) 2-B(1) 3-A(2) 4-B(2) ... 19-A(10) 20-B(10)

Thus the average time for both was 19+20/2 = 19.5.

If the scheduling granularity was set to 10, it would go like:

1-A(1) 2-A(2) 3-A(3) ... 10-A(10) 11-B(1) 12-B(2) ... 20-B(10)

Thus the average time for both was 10+20 / 2 = 15.  So curiously 
enough, letting a single thread complete first LOWERED the overall 
latency of the system.   Also note, the cost for switching contexts was 
FREE... lets say you paid a penalty of 1 tick to switch threads (100% 
overhead for the 1 tick case) you'd see a ratio of (19+20+19) / 2 = 29 
and (10+20+1) / 2 = 15.5  -- A big difference!  Obviously this is a 
somewhat contrived example, since you don't know in advance how long it 
will take to serve a Zope request, but I hope it should make you feel 
better about being able to turn down the number of Zope threads without 
losing anything.

On May 17, 2005, at 1:15 PM, David Diskin wrote:
[...][...]