Hello all, pardon an interruption for wannabe Deep Thoughts(tm)... took 
a while to get around to composing the mail,

Here are two small products for review before placing them online 
somewhere. A need of mine when programming in Zope is finding a fast 
(to code) and persistent storage area in the ZODB to manipulate from a 
combination of python script and ZPT. When saving things like user 
preferences, state, simple mappings of items, buffers, or miscellaneous 
things there are not many choices i've found... A DB for it is a bit 
much, you can use properties, but these do not support true 
dictionaries or lists (lists having values of strings only, etc., 
apparently?) and are mated to the folder, and it's pretty general for 
even an archetype... I've used less elegant methods to store for quick 
testing and rollout. Regardless, monolithic Plone Content Type storage 
can offer more integrated searching and cataloging - worth the effort, 
sometimes - but for when those things don't matter, and programming 
does, I present: Hash and List.

My m.i.m.b.y (made in my backyard, literally :) ) approach is attached. 
To install just place in the products directory. When adding, specify 
it's ID and optionally, the title. A reference to it's methods is 
available right in the ZMI when you add or look at the object. It will 
also report it's contents in this view. Note that by only letting 
methods access the internal storage we get transactional undo and 
standard security settings on these objects. You should probably copy 
it's contents to a local variable and re update the contents of the 
object at the end of logical blocks or the end of the script, as i'm 
not sure if it could add to the ZODB rapidly if called in loops, etc? 
I'm not entirely certain when and on what the ZODB commits transactions 
- perhaps on each method call? - but this works fine anyway.

Think of it as a a new friend of the python script type. If you have 
use for this, or just have comments, please air them. If you know a 
better way, I won't be too upset to hear it, either. Thanks for taking 
the time to look at and hopefully test this.

Happy Zoping!

         James Robey

[...]

James --

I've only glanced at your implementation, so don't take this as a review
of your code, but of your idea.

In general, these seem like fine classes. However, I'm not sure this is
following good design for Zope/Plone.

You don't want an "application" to be a bunch of very-loosely-coupled
DTML/PythonScripts/ZPTs. Writing a product that just acts like a "general
data container" might be useful, but I don't think it's a good path to
head down.

For example, let's say you're writing a student-registration product. You
*could* use your Hash class to keep track of data about a class, since you
can store that kind of stuff in a dict. But much, much better is to create
a SimpleItem object that represents the class, and provide APIs on that
for the things like getClassTitle, etc. Once you've done that once or
twice, you can get that down to 5 minutes, and you end up with something
*much* more maintainable and organized. Plus, then, you don't have to
re-create a pseudo-list type or pseudo-hash type, but can simple store
this information as a real, normal list or or hash in your object.
Then,you end up with your data object being bound to the type of data it
is, and it can have methods, etc., rather than just a dumb container of
data with logic and presentation splayed elsewhere.

Or, of course, if you're making something contentish, use Archetypes,
which can do this an make it even simpler. With things like ArchGenXML,
you can even do that without typing a line of Python.

Perhaps for very throwaway stuff, you might find this useful. But if I was
doing somethat that throwaway, I'd just do the classic throwaway Zope
hack, and create something (folder? dtml method? it doesn't matter) and
add my list or dict (or BTree, see below) onto that directly (perfectly
legal, tho' it must be done in trusted code). Then my scripts/skins can
use that hanging-on data. I don't recommend this, but sometimes it can be
useful for this kind of
you-need-it-five-minutes-from-now-and-it's-just-for-your-personal-site
kind of thing.

Hope you find this helpful.

Cheers,

- j.

P.S. If you really want to create a Hash class, I'd read up on BTrees.
Storing a pure dict in the ZODB is a Bad Idea unless it's going to remain
quite small. Perhaps BTrees would be a good subject for a future ZPUGDC
meeting...

echo that. it's a discussion on storage (and overhead of it) whose  
parameters i have trouble defining, but i want to have nonetheless. I  
AM interested in rapid protoyping of these ideas so that i can try lots  
of ideas and let darwin do the rest..

Also, my product doesn't work, exactly, and it has to do with  
committing a transaction to that product. In one instance, i could  
refresh the browser over and over, and sometimes it would find the keys  
and values of the Hash, at other times it would read empty! Alternating  
back and forth, really spooky. Anyway, i think i'll spend more time  
with this.

It may be (in the end) that i find I continue to build my idea and end  
up with... archetypes. But at least I learned :)

thanks all, and I wouldn't use what i put out quite yet, but i'd be up  
on someone looking over that code? c'mon! help-a-brotha-out... ah, i  
tried.

james.
On Apr 19, 2005, at 10:18 AM, Joel Burton wrote:
[...][...][...]

You're probably discovering the bad stuff that can happen when you  
write module-level data that isn't transactionally aware.

In a nutshell -- don't!

You can only do it if you *really* *really* know what you're doing.   
Once you step out of Zope's transaction machinery you'll have to  
maintain data consistency yourself, by registering with Zope's  
transaction monitor and receiving notices of votes, commits, and  
aborts.

While this isn't terribly hard, it isn't really worth it most of the  
time.

I suspect a brief discussion of "What's a transaction" and "What's the  
ZODB" are in order at some point in time, as they relate to each other.  
  To be brutally brief, the ZODB consists of a set of transaction  
records, each of which contain object records.   The transaction  
manager and the ZODB work together to provide a consistent object view;  
however, conflicts can and do occur.   If transaction A: (Add 1 to X)  
runs simultaneously with transaction B: (Add 1 to X) one of the two is  
going to get a conflict.  Because each object has a transaction  
associated with its current state, the ZODB can know if another  
transaction has invalidated that object at commit time, e.g.  
Transaction A gets X.20050419135700 and transaction B gets  
X.20050419135700, A commits, creating X.20050419135800, when B commits  
the version of X it has is no longer current, so a WriteConflict  
occurs.

Also, consider that the ZODB writes "objects" not "data structures."   
As such, very large objects which are mutable are strongly discouraged.  
  This is why Zope has something called a "BTree" (I dislike the name  
because Zope BTrees aren't height balanced, which is what a Computer  
Science B-Tree is) -- a dictionary-like object which segments its space  
into separate object buckets.  Each bucket can be mutated without  
requiring a re-write of the entire tree.  This STILL can be terribly  
inefficient -- the Zope Catalog usually consumes about 90% of the  
typical ZODB.

On Apr 19, 2005, at 11:04 AM, jamesr wrote:
[...]
>>> Here are two small products for review before placing them online
>>> somewhere. A need of mine when programming in Zope is finding a
fast
>>> (to code) and persistent storage area in the ZODB to manipulate
from  
>>> a
>>> combination of python script and ZPT. When saving things like user
>>> preferences, state, simple mappings of items, buffers, or  
>>> miscellaneous
>>> things there are not many choices i've found... A DB for it is a
bit
>>> much, you can use properties, but these do not support true
>>> dictionaries or lists (lists having values of strings only, etc.,
>>> apparently?) and are mated to the folder, and it's pretty general
for
>>> even an archetype... I've used less elegant methods to store for  
>>> quick
>>> testing and rollout. Regardless, monolithic Plone Content Type  
>>> storage
>>> can offer more integrated searching and cataloging - worth the  
>>> effort,
>>> sometimes - but for when those things don't matter, and
programming
>>> does, I present: Hash and List.
>>>
>>> My m.i.m.b.y (made in my backyard, literally :) ) approach is  
>>> attached.
>>> To install just place in the products directory. When adding,
specify
>>> it's ID and optionally, the title. A reference to it's methods is
>>> available right in the ZMI when you add or look at the object. It 

>>> will
>>> also report it's contents in this view. Note that by only letting
>>> methods access the internal storage we get transactional undo and
>>> standard security settings on these objects. You should probably
copy
>>> it's contents to a local variable and re update the contents of
the
>>> object at the end of logical blocks or the end of the script, as
i'm
>>> not sure if it could add to the ZODB rapidly if called in loops,
etc?
>>> I'm not entirely certain when and on what the ZODB commits  
>>> transactions
>>> - perhaps on each method call? - but this works fine anyway.
>>>
>>> Think of it as a a new friend of the python script type. If you
have
>>> use for this, or just have comments, please air them. If you know
a
>>> better way, I won't be too upset to hear it, either. Thanks for  
>>> taking
>>> the time to look at and hopefully test this.[...][...]

[...]

Do you understand that you have to use either Zope's special
persistance-aware dicts and lists, or that you have to tickle the
_p_changed magical attribute? It sounds like perhaps you're not doing
those. (Or, as mentioned before, ditch the dict and use a BTree).

You can find a good discussion of this in the Zope Developers Guide (tho'
that's a somewhat outdated doc, there's good stuff in it).

- j.

i made a brief sweep over the installed code and found  
get_transaction.commit() - although i couldn't find what namespace that  
code was living at in the short time I looked for it. buuuut,  
_p_changed is just the attribute I'm looking for! Thanks,  
experimentation is required. I'll learn what i can about efficiency  
meanwhile, I haven't given up the sport of it.

I believe that was even mentioned in the very first meeting of the new  
cycle. Viva la zpug.


On Apr 19, 2005, at 2:45 PM, Joel Burton wrote:
[...][...][...]