[MTOS-dev] MT::Object::load_iter() problem

[Hirotaka Ogawa]

On Thu, Mar 6, 2008 at 3:42 AM, Brad Choate <brad at sixapart.com> wrote:
> Ogawa-san-- thanks for your continued work and feedback in this area.
>  We're actively working on this as well, and I just realized two other
>  places we stash things that may account for the remaining delta you're
>  seeing.

>  In MT::Object, there is a 'cache_property' method that is used
>  to cache data in the object itself. I'm moving that cache storage into
>  MT::Request, and will be committing that change soon into the
>  release-31 branch.

I'm not sure this part.  Do you mean that cache_property had been
bundled with MT::Object as a field, but now was moved to MT::Request?

So, cache_property would be able to be cleared safely and surely even
if its parent MT::Object was leaked.

>  Also, when we look for the previous/next entry on a
>  given entry object, a reference to that object is stored in the entry.
>  This could be weakened, but I would prefer just storing the id-- the
>  object is already cached in RAM, so keeping the id should be good
>  enough.

>From view point of time-space tradeoffs, I think this is reasonable.

>
>  -Brad
>
>
>
>
>  On Mar 5, 2008, at 2:03 AM, Hirotaka Ogawa wrote:
>
>  > MT::Object::load_iter() is really complicated.
>  >
>  > When ObjectDriver is DBI::sqlite, load_iter() fetches objects from
>  > MT::OD::D::Cache::RAM, which looks up the internal cache first and
>  > queries the database only when lookup failed.
>  >
>  > Otherwise, load_iter() fetches objects from MT::OD::D::DBI which
>  > directly queries the database, and then stores fetched objects to
>  > MT::OD::D::Cache::RAM, internal cache.  That is, when using mysql or
>  > postgres, cached objects stored by a load_iter() are never reused by
>  > the next load_iter() requests.  They could be reused only if they were
>  > required by the next load() methods.
>  >
>  > In addition, objects loaded by load_iter() seem to be leaked.  After
>  > calling $MT::Object::DRIVER->clear_cache() which clears all contents
>  > of the internal cache, some objects are never destroyed and still
>  > remain in memory.
>  >
>  > To fix this situation, load_iter() fetches objects from
>  > MT::OD::D::Cache::RAM[1] or MT::OD::D::DBI[2], without depending on
>  > ObjectDriver setting.
>  >
>  > [1] Using MT::OD::D::Cache::RAM (same as DBI::sqlite case)
>  >
>  > sub load_iter   {
>  >    my $class = shift;
>  >    my $driver = $class->driver;
>  >    return scalar $driver->search($class, @_);
>  > }
>  >
>  > [2] Using MT::OD::D::DBI
>  >
>  > sub load_iter   {
>  >    my $class = shift;
>  >    my $driver = $class->driver;
>  >    while ( $driver->isa('Data::ObjectDriver::Driver::BaseCache') ) {
>  >        $driver = $driver->fallback;
>  >    }
>  >    return scalar $driver->search($class, @_);
>  > }
>  >
>  > But, after applying this modification, still leak remains.
>  >
>  > Using FCGI environment, I reapeatedly performed 'rebuild all' for a
>  > relatively small blog with 237 entries, 390 comments, and 1745
>  > trackbacks and measured RSS of the FCGI proccess.
>  >
>  > Without weaken:
>  >
>  > Initial       25,152KB
>  > 1st rebuild   60,192KB
>  > 2nd rebuild   87,144KB
>  > 3rd rebuild  114,160KB
>  > 4th rebuild  140,848KB
>  > 5th rebuild  163,164KB
>  >
>  > With weaken:
>  >
>  > Initial       24,544KB
>  > 1st rebuild   40,332KB
>  > 2nd rebuild   46,388KB
>  > 3rd rebuild   52,424KB
>  > 4th rebuild   58,464KB
>  > 5th rebuild   64,524KB
>  >
>  > With weaken + load_iter mods:
>  >
>  > Initial       24,528KB
>  > 1st rebuild   38,032KB
>  > 2nd rebuild   42,544KB
>  > 3rd rebuild   46,908KB
>  > 4th rebuild   51,204KB
>  > 5th rebuild   55,708KB
>  >
>  > --
>  > Hirotaka Ogawa makes no sense.
>  > http://as-is.net/blog/
>  > _______________________________________________
>  > MTOS-dev mailing list
>  > MTOS-dev at sixapart.com
>  > http://www.sixapart.com/mailman/listinfo/mtos-dev
>
>  --
>  Brad Choate
>  Engineering Manager, Movable Type, Six Apart, Ltd.
>  http://www.sixapart.com/movabletype/
>  Mobile: (918) 271-0105  -  AIM: bschoate
>
>
>
>
>



-- 
Hirotaka Ogawa makes no sense.
http://as-is.net/blog/