[MTOS-dev] MT::Object::load_iter() problem

[Hirotaka Ogawa]

On Thu, Mar 6, 2008 at 3:42 AM, Brad Choate <brad at sixapart.com> wrote:
> Ogawa-san-- thanks for your continued work and feedback in this area.
>  We're actively working on this as well, and I just realized two other
>  places we stash things that may account for the remaining delta you're
>  seeing. In MT::Object, there is a 'cache_property' method that is used
>  to cache data in the object itself. I'm moving that cache storage into
>  MT::Request, and will be committing that change soon into the
>  release-31 branch. Also, when we look for the previous/next entry on a
>  given entry object, a reference to that object is stored in the entry.
>  This could be weakened, but I would prefer just storing the id-- the
>  object is already cached in RAM, so keeping the id should be good
>  enough.

Okay.  I'll continue watching around this area attentively.

FYI, after I modified load_iter() as I posted, it seemed that the
count of blessed objects never grew, but memory consumption still
grew.  This fact indicates that there're circular references or
something in between unblessed objects.

And, I'd like to propose that load_iter should create a kind of
"Iterator" object and return this object, instead of a closure.  If
so, we can easily observe how Iterators are created, destroyed, and
leaked, also we can add utility methods such as 'finish'.

>
>  -Brad
>
>
>
>
>  On Mar 5, 2008, at 2:03 AM, Hirotaka Ogawa wrote:
>
>  > MT::Object::load_iter() is really complicated.
>  >
>  > When ObjectDriver is DBI::sqlite, load_iter() fetches objects from
>  > MT::OD::D::Cache::RAM, which looks up the internal cache first and
>  > queries the database only when lookup failed.
>  >
>  > Otherwise, load_iter() fetches objects from MT::OD::D::DBI which
>  > directly queries the database, and then stores fetched objects to
>  > MT::OD::D::Cache::RAM, internal cache.  That is, when using mysql or
>  > postgres, cached objects stored by a load_iter() are never reused by
>  > the next load_iter() requests.  They could be reused only if they were
>  > required by the next load() methods.
>  >
>  > In addition, objects loaded by load_iter() seem to be leaked.  After
>  > calling $MT::Object::DRIVER->clear_cache() which clears all contents
>  > of the internal cache, some objects are never destroyed and still
>  > remain in memory.
>  >
>  > To fix this situation, load_iter() fetches objects from
>  > MT::OD::D::Cache::RAM[1] or MT::OD::D::DBI[2], without depending on
>  > ObjectDriver setting.
>  >
>  > [1] Using MT::OD::D::Cache::RAM (same as DBI::sqlite case)
>  >
>  > sub load_iter   {
>  >    my $class = shift;
>  >    my $driver = $class->driver;
>  >    return scalar $driver->search($class, @_);
>  > }
>  >
>  > [2] Using MT::OD::D::DBI
>  >
>  > sub load_iter   {
>  >    my $class = shift;
>  >    my $driver = $class->driver;
>  >    while ( $driver->isa('Data::ObjectDriver::Driver::BaseCache') ) {
>  >        $driver = $driver->fallback;
>  >    }
>  >    return scalar $driver->search($class, @_);
>  > }
>  >
>  > But, after applying this modification, still leak remains.
>  >
>  > Using FCGI environment, I reapeatedly performed 'rebuild all' for a
>  > relatively small blog with 237 entries, 390 comments, and 1745
>  > trackbacks and measured RSS of the FCGI proccess.
>  >
>  > Without weaken:
>  >
>  > Initial       25,152KB
>  > 1st rebuild   60,192KB
>  > 2nd rebuild   87,144KB
>  > 3rd rebuild  114,160KB
>  > 4th rebuild  140,848KB
>  > 5th rebuild  163,164KB
>  >
>  > With weaken:
>  >
>  > Initial       24,544KB
>  > 1st rebuild   40,332KB
>  > 2nd rebuild   46,388KB
>  > 3rd rebuild   52,424KB
>  > 4th rebuild   58,464KB
>  > 5th rebuild   64,524KB
>  >
>  > With weaken + load_iter mods:
>  >
>  > Initial       24,528KB
>  > 1st rebuild   38,032KB
>  > 2nd rebuild   42,544KB
>  > 3rd rebuild   46,908KB
>  > 4th rebuild   51,204KB
>  > 5th rebuild   55,708KB
>  >
>  > --
>  > Hirotaka Ogawa makes no sense.
>  > http://as-is.net/blog/
>  > _______________________________________________
>  > MTOS-dev mailing list
>  > MTOS-dev at sixapart.com
>  > http://www.sixapart.com/mailman/listinfo/mtos-dev
>
>  --
>  Brad Choate
>  Engineering Manager, Movable Type, Six Apart, Ltd.
>  http://www.sixapart.com/movabletype/
>  Mobile: (918) 271-0105  -  AIM: bschoate
>
>
>
>
>



-- 
Hirotaka Ogawa makes no sense.
http://as-is.net/blog/