[mlpack] Memory usage during build

[Markus Kolb]

Hello,

do you know about the "high" memory usage during build of mlpack?
With version 3.3.2 a machine with guaranteed physical memory of 4.8G 
when using 2 CPUs has been enough.
Now when compiling 3.4.2 a machine with 8G physical memory and using 2 
CPUs is going in OOM.

[ 5447s] cd /home/abuild/rpmbuild/BUILD/mlpack-3.4.2/build && 
/usr/bin/cmake -E cmake_depends "Unix Makefiles" 
/home/abuild/rpmbuild/BUILD/mlpack-3.4.2 
/home/abuild/rpmbuild/BUILD/mlpack-3.4.2/src/mlpack 
/home/abuild/rpmbuild/BUILD/mlpack-3.4.2/build 
/home/abuild/rpmbuild/BUILD/mlpack-3.4.2/build/src/mlpack 
/home/abuild/rpmbuild/BUILD/mlpack-3.4.2/build/src/mlpack/CMakeFiles/python_configure.dir/DependInfo.cmake
[ 5447s] Scanning dependencies of target python_configure
[ 5447s] make[2]: Leaving directory 
'/home/abuild/rpmbuild/BUILD/mlpack-3.4.2/build'
[ 5447s] /usr/bin/make -f 
src/mlpack/CMakeFiles/python_configure.dir/build.make 
src/mlpack/CMakeFiles/python_configure.dir/build
[ 5447s] make[2]: Entering directory 
'/home/abuild/rpmbuild/BUILD/mlpack-3.4.2/build'
[ 5447s] [ 39%] Configuring setup.py...
[ 5447s] cd /home/abuild/rpmbuild/BUILD/mlpack-3.4.2/build/src/mlpack && 
/usr/bin/cmake -D 
SETUP_PY_IN=/home/abuild/rpmbuild/BUILD/mlpack-3.4.2/src/mlpack/bindings/python/setup.py.in 
-D 
SETUP_PY_OUT=/home/abuild/rpmbuild/BUILD/mlpack-3.4.2/build/src/mlpack/bindings/python/setup.py 
-D PACKAGE_VERSION="3.4.2" -D 
Boost_SERIALIZATION_LIBRARY="/usr/lib64/libboost_serialization.so" -D 
Boost_LIBRARY_DIRS="/usr/lib64" -D 
ARMADILLO_LIBRARIES="/usr/lib64/libarmadillo.so" -D 
MLPACK_LIBRARY=/home/abuild/rpmbuild/BUILD/mlpack-3.4.2/build/lib/libmlpack.so 
-D MLPACK_LIBDIR=/home/abuild/rpmbuild/BUILD/mlpack-3.4.2/build/lib -D 
MLPACK_PYXS="arma_numpy.pyx test_python_binding.pyx adaboost.pyx 
approx_kfn.pyx bayesian_linear_regression.pyx cf.pyx dbscan.pyx 
decision_stump.pyx decision_tree.pyx det.pyx emst.pyx fastmks.pyx 
gmm_train.pyx gmm_generate.pyx gmm_probability.pyx hmm_train.pyx 
hmm_loglik.pyx hmm_viterbi.pyx hmm_generate.pyx hoeffding_tree.pyx 
kde.pyx kernel_pca.pyx kmeans.pyx lars.pyx linear_regression.pyx 
linear_svm.pyx lmnn.pyx local_coordinate_coding.pyx 
logistic_regression.pyx lsh.pyx mean_shift.pyx nbc.pyx nca.pyx knn.pyx 
kfn.pyx nmf.pyx pca.pyx perceptron.pyx preprocess_split.pyx 
preprocess_binarize.pyx preprocess_describe.pyx preprocess_scale.pyx 
preprocess_one_hot_encoding.pyx image_converter.pyx radical.pyx 
random_forest.pyx krann.pyx softmax_regression.pyx sparse_coding.pyx" -D 
OpenMP_CXX_FLAGS="-fopenmp" -D DISABLE_CFLAGS="" -D 
CYTHON_INCLUDE_DIRECTORIES="/home/abuild/rpmbuild/BUILD/mlpack-3.4.2/src 
/usr/include /usr/include /usr/include/stb /usr/include 
/home/abuild/rpmbuild/BUILD/mlpack-3.4.2/build/src/mlpack/.." -D 
CMAKE_CXX_FLAGS=-fmessage-length=0\ -grecord-gcc-switches\ -O2\ -Wall\ 
-D_FORTIFY_SOURCE=2\ -fstack-protector-strong\ -funwind-tables\ 
-fasynchronous-unwind-tables\ -fstack-clash-protection\ -DNDEBUG\ -Wall\ 
-Wextra\ -ftemplate-depth=1000\ -Wno-unused-function\ -O3\ -fopenmp -D 
OUTPUT_DIR=/home/abuild/rpmbuild/BUILD/mlpack-3.4.2/build -P 
/home/abuild/rpmbuild/BUILD/mlpack-3.4.2/src/mlpack/bindings/python/ConfigureSetup.cmake
[ 5447s] -- Run with /usr/lib64/libboost_serialization.so.
[ 5447s] make[2]: Leaving directory 
'/home/abuild/rpmbuild/BUILD/mlpack-3.4.2/build'
[ 5447s] make[1]: Entering directory 
'/home/abuild/rpmbuild/BUILD/mlpack-3.4.2/build'
[ 5447s] [ 39%] Built target python_configure
[ 5447s] make[1]: Leaving directory 
'/home/abuild/rpmbuild/BUILD/mlpack-3.4.2/build'
[ 6994s] [ 6977.215743] Out of memory: Killed process 10354 (cc1plus) 
total-vm:8193592kB, anon-rss:5490544kB, file-rss:0kB, shmem-rss:0kB
[ 6995s] make[2]: Entering directory 
'/home/abuild/rpmbuild/BUILD/mlpack-3.4.2/build'
[ 6995s] [ 38%] Building CXX object 
src/mlpack/tests/CMakeFiles/mlpack_catch_test.dir/feedforward_network_test.cpp.o
[ 6995s] cd 
/home/abuild/rpmbuild/BUILD/mlpack-3.4.2/build/src/mlpack/tests && 
/usr/bin/c++  -DARMA_NO_DEBUG -DBOOST_TEST_DYN_LINK -DHAS_OPENMP 
-DHAS_STB -DNDEBUG -I/home/abuild/rpmbuild/BUILD/mlpack-3.4.2/src 
-I/usr/include/stb 
-I/home/abuild/rpmbuild/BUILD/mlpack-3.4.2/build/src/mlpack/..  
-fmessage-length=0 -grecord-gcc-switches -O2 -Wall -D_FORTIFY_SOURCE=2 
-fstack-protector-strong -funwind-tables -fasynchronous-unwind-tables 
-fstack-clash-protection -DNDEBUG -Wall -Wextra -ftemplate-depth=1000 
-Wno-unused-function -O3 -fopenmp -O2 -g -DNDEBUG   -Winvalid-pch 
-include 
/home/abuild/rpmbuild/BUILD/mlpack-3.4.2/build/src/mlpack/tests/cotire/mlpack_catch_test_CXX_prefix.hxx 
-std=gnu++11 -o 
CMakeFiles/mlpack_catch_test.dir/feedforward_network_test.cpp.o -c 
/home/abuild/rpmbuild/BUILD/mlpack-3.4.2/src/mlpack/tests/feedforward_network_test.cpp
[ 6995s] c++: internal compiler error: Killed (program cc1plus)
[ 6995s] make[2]: *** 
[src/mlpack/tests/CMakeFiles/mlpack_catch_test.dir/build.make:518: 
src/mlpack/tests/CMakeFiles/mlpack_catch_test.dir/feedforward_network_test.cpp.o] 
Error 4
[ 6995s] make[2]: Leaving directory 
'/home/abuild/rpmbuild/BUILD/mlpack-3.4.2/build'
[ 6995s] make[2]: *** Waiting for unfinished jobs....



What is being done here during build that needs so much memory?
In relation to the size of mlpack library this looks strange.
Is the memory usage ok or is there anything wrong in the build process?

Thanks
Markus