[mlpack] Memory usage during build

[Ryan Curtin]

On Thu, Dec 10, 2020 at 02:03:44PM +0100, Markus Kolb wrote:
> Hello,
> 
> do you know about the "high" memory usage during build of mlpack?
> With version 3.3.2 a machine with guaranteed physical memory of 4.8G when
> using 2 CPUs has been enough.
> Now when compiling 3.4.2 a machine with 8G physical memory and using 2 CPUs
> is going in OOM.
>
> ... 
> 
> What is being done here during build that needs so much memory?
> In relation to the size of mlpack library this looks strange.
> Is the memory usage ok or is there anything wrong in the build process?

Hi Markus,

Unfortunately it seems that over the years, C++ compilers have become
more and more hungry for memory.  So that is part of the issue.  I'm
surprised at your report that 8GB memory can't build on 2 cores, but I
can't think of anything that would be wrong there.  You could try seeing
if something else is using RAM too during the build?  There are no
significant code changes between 3.3.2 and 3.4.2 that should cause
memory blowups; I wonder if the version of the compiler makes a
difference here.

That said, we are investigating ways to accelerate our builds and figure
out how we can reduce the memory footprint.  We've been pushing towards
removing Boost as a dependency, and I think that this will be very
helpful.

Hope this helps!

Thanks,

Ryan

-- 
Ryan Curtin    | "What? Facts?"
ryan at ratml.org |   - Joe Cairo