[MTOS-dev] MT::Object::load_iter() problem

[Brad Choate]

Ogawa-san-- thanks for your continued work and feedback in this area.  
We're actively working on this as well, and I just realized two other  
places we stash things that may account for the remaining delta you're  
seeing. In MT::Object, there is a 'cache_property' method that is used  
to cache data in the object itself. I'm moving that cache storage into  
MT::Request, and will be committing that change soon into the  
release-31 branch. Also, when we look for the previous/next entry on a  
given entry object, a reference to that object is stored in the entry.  
This could be weakened, but I would prefer just storing the id-- the  
object is already cached in RAM, so keeping the id should be good  
enough.

-Brad


On Mar 5, 2008, at 2:03 AM, Hirotaka Ogawa wrote:

> MT::Object::load_iter() is really complicated.
>
> When ObjectDriver is DBI::sqlite, load_iter() fetches objects from
> MT::OD::D::Cache::RAM, which looks up the internal cache first and
> queries the database only when lookup failed.
>
> Otherwise, load_iter() fetches objects from MT::OD::D::DBI which
> directly queries the database, and then stores fetched objects to
> MT::OD::D::Cache::RAM, internal cache.  That is, when using mysql or
> postgres, cached objects stored by a load_iter() are never reused by
> the next load_iter() requests.  They could be reused only if they were
> required by the next load() methods.
>
> In addition, objects loaded by load_iter() seem to be leaked.  After
> calling $MT::Object::DRIVER->clear_cache() which clears all contents
> of the internal cache, some objects are never destroyed and still
> remain in memory.
>
> To fix this situation, load_iter() fetches objects from
> MT::OD::D::Cache::RAM[1] or MT::OD::D::DBI[2], without depending on
> ObjectDriver setting.
>
> [1] Using MT::OD::D::Cache::RAM (same as DBI::sqlite case)
>
> sub load_iter   {
>    my $class = shift;
>    my $driver = $class->driver;
>    return scalar $driver->search($class, @_);
> }
>
> [2] Using MT::OD::D::DBI
>
> sub load_iter   {
>    my $class = shift;
>    my $driver = $class->driver;
>    while ( $driver->isa('Data::ObjectDriver::Driver::BaseCache') ) {
>        $driver = $driver->fallback;
>    }
>    return scalar $driver->search($class, @_);
> }
>
> But, after applying this modification, still leak remains.
>
> Using FCGI environment, I reapeatedly performed 'rebuild all' for a
> relatively small blog with 237 entries, 390 comments, and 1745
> trackbacks and measured RSS of the FCGI proccess.
>
> Without weaken:
>
> Initial       25,152KB
> 1st rebuild   60,192KB
> 2nd rebuild   87,144KB
> 3rd rebuild  114,160KB
> 4th rebuild  140,848KB
> 5th rebuild  163,164KB
>
> With weaken:
>
> Initial       24,544KB
> 1st rebuild   40,332KB
> 2nd rebuild   46,388KB
> 3rd rebuild   52,424KB
> 4th rebuild   58,464KB
> 5th rebuild   64,524KB
>
> With weaken + load_iter mods:
>
> Initial       24,528KB
> 1st rebuild   38,032KB
> 2nd rebuild   42,544KB
> 3rd rebuild   46,908KB
> 4th rebuild   51,204KB
> 5th rebuild   55,708KB
>
> -- 
> Hirotaka Ogawa makes no sense.
> http://as-is.net/blog/
> _______________________________________________
> MTOS-dev mailing list
> MTOS-dev at sixapart.com
> http://www.sixapart.com/mailman/listinfo/mtos-dev

--
Brad Choate
Engineering Manager, Movable Type, Six Apart, Ltd.
http://www.sixapart.com/movabletype/
Mobile: (918) 271-0105  -  AIM: bschoate