run.sh fails on fresh 12.04 Ubuntu

[jamesd256]

Hi,

Thanks for your efforts, I really hope I can get further with your work as it looks promising.

I get the following error. I have tried installing some blas related libraries with apt (groping in the dark? Openblas and libblass stuff), but can't get past this:

/tmp/ccBgO7xE.o: In function add_numpy_builtin': hello.c:(.text+0x5f): undefined reference toinit_dotblas'
collect2: ld returned 1 exit status

I also gave it a shot having installed numpy via pip, with the same result.

Thanks,

James

[patmarion]

Ah! The README should be updated to explain this. If blas and lapack are installed on the system, then numpy will be configured with blas and lapack support. That adds a required link dependency on those libraries. Edit run.sh and you'll find a variable named requiredBlasLibrary. For macosx, it is set to a default value, but for linux it is empty. You have to fill that in for your system. I just tested it on Ubuntu 14, and here is the value that worked for me:

requiredBlasLibrary="/usr/lib/liblapack.a /usr/lib/libblas.a /usr/lib/gcc/x86_64-linux-gnu/4.8/libgfortran.a"

By the way, in run.sh you can comment out the lines at the bottom so that the script only re-runs the linking, and that way you don't have to re-compile python and numpy each time you run the script. So it would look like:

#extractAndBuildPython
#extractAndBuildNumpy
#runFreezeTool
buildHelloNumpy

For more information about linkage, see:

http://stackoverflow.com/questions/9000164/how-to-check-blas-lapack-linkage-in-numpy-scipy

You can also check the dynamic linkage on the compiled numpy modules in the build output directory, for example:

$ ldd ./build/numpy-1.6.2/build/lib.linux-x86_64-2.7/numpy/core/_dotblas.so
    linux-vdso.so.1 =>  (0x00007ffe451f9000)
    libcblas.so.3 => /usr/lib/libcblas.so.3 (0x00007ff10b542000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007ff10b324000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ff10af5f000)
    libatlas.so.3 => /usr/lib/libatlas.so.3 (0x00007ff10a9cc000)
    libgfortran.so.3 => /usr/lib/x86_64-linux-gnu/libgfortran.so.3 (0x00007ff10a6b2000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007ff10a49c000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007ff10a196000)
    /lib64/ld-linux-x86-64.so.2 (0x00007ff10b969000)
    libquadmath.so.0 => /usr/lib/x86_64-linux-gnu/libquadmath.so.0 (0x00007ff109f5a000)

$ ldd ./build/numpy-1.6.2/build/lib.linux-x86_64-2.7/numpy/linalg/lapack_lite.so
    linux-vdso.so.1 =>  (0x00007ffee099f000)
    liblapack.so.3 => /usr/lib/liblapack.so.3 (0x00007f609193f000)
    libf77blas.so.3 => /usr/lib/libf77blas.so.3 (0x00007f609171f000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f609135a000)
    libblas.so.3 => /usr/lib/libblas.so.3 (0x00007f6090d8e000)
    libgfortran.so.3 => /usr/lib/x86_64-linux-gnu/libgfortran.so.3 (0x00007f6090a74000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f609076e000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f6090558000)
    libcblas.so.3 => /usr/lib/libcblas.so.3 (0x00007f6090337000)
    libatlas.so.3 => /usr/lib/libatlas.so.3 (0x00007f608fda4000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f608fb86000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f60922e3000)
    libquadmath.so.0 => /usr/lib/x86_64-linux-gnu/libquadmath.so.0 (0x00007f608f94a000)

Other libraries you might need to add are /usr/lib/libcblas.a and /usr/lib/libatlas.a, and note that the order that .a library files are listed does matter.

[patmarion]

Another project you may be interested in is Slither. Whereas this project, NumpyBuiltinExample attempts to provide the minimal example, Slither is an automated tool to help you build frozen programs with numpy: https://github.com/bfroehle/slither

[patmarion]

Oh wait. Reading your error message, it looks like you don't have the init_dotblas symbol, and that means you don't have blas and lapack installed, so numpy was configured without it. You probably don't have this file either:

build/numpy-1.6.2/build/lib.linux-x86_64-2.7/numpy/core/_dotblas.so

So, you can either install blas and then re-run the build, or you can remove the reference to init_dotblas from files/helloNumpy/numpy_builtin.h. Edit that file and comment out the two lines where dotblas is mentioned.

[jamesd256]

Thanks for the updates. I have looked over the slither project, which also looks very interesting.

However, in the interest of completing my test with NumpyBuiltinExample, I also went ahead with some of your suggestions, so if it is of use to you to continue supporting!

I installed blas and lapack, and also gfortran.

sudo apt-get install libblas-dev liblapack-dev gfortran

I checked that my requiredBlasLibrary libs exist:

$ ls -al /usr/lib/liblapack.a /usr/lib/libblas.a /usr/lib/gcc/x86_64-linux-gnu/4.6/libgfortran.a
-rw-r--r-- 1 root root 2700976 Apr 15 2012 /usr/lib/gcc/x86_64-linux-gnu/4.6/libgfortran.a
lrwxrwxrwx 1 root root 27 Apr 3 20:18 /usr/lib/libblas.a -> /etc/alternatives/libblas.a
lrwxrwxrwx 1 root root 29 Apr 4 08:01 /usr/lib/liblapack.a -> /etc/alternatives/liblapack.a

Unfortunately I got a similar result:

/tmp/cc226X55.o: In functionadd_numpy_builtin': hello.c:(.text+0x5f): undefined reference to 'init_dotblas' collect2: ld returned 1 exit status`

I also tried with the dotblas references commented out in the numpy_builtin.h as suggested, with this result:

In file included from hello.c:4:0:
numpy_builtin.h:14:2: error: invalid preprocessing directive #extern
numpy_builtin.h: In function ‘add_numpy_builtin’:
numpy_builtin.h:28:4: error: invalid preprocessing directive #PyImport_AppendInittab

[jamesd256]

Also, I tried to look at the linking dependencies, but there is no _dotblas.so generated in build.

I studied the output of run.sh, and found this section which may be of interest:

building extension "numpy.core._dotblas" sources
building extension "numpy.core.umath_tests" sources
conv_template:> build/src.linux-x86_64-2.7/numpy/core/src/umath/umath_tests.c
building extension "numpy.core.multiarray_tests" sources
conv_template:> build/src.linux-x86_64-2.7/numpy/core/src/multiarray/multiarray_tests.c
building extension "numpy.lib._compiled_base" sources
building extension "numpy.numarray._capi" sources
building extension "numpy.fft.fftpack_lite" sources
building extension "numpy.linalg.lapack_lite" sources
creating build/src.linux-x86_64-2.7/numpy/linalg
  adding 'numpy/linalg/lapack_litemodule.c' to sources.
  adding 'numpy/linalg/python_xerbla.c' to sources.
building extension "numpy.random.mtrand" sources
creating build/src.linux-x86_64-2.7/numpy/random
C compiler: gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC
compile options: '-Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/cor
e/include -I/home/jamesjdonnelly/NumpyBuiltinExample-master/build/install/include/python2.7 -c'
gcc: _configtest.c
gcc -pthread _configtest.o -o _configtest
_configtest
failure.

[patmarion]

Comment out lines using //, not #.

[jamesd256]

Wow, did I do that? Obviously forgot I was editing a cpp header. :(

Anyway, yes, commenting out those header lines does the job, and I can build hello and helloFrozen, run the test.py and confirm that the module loading is as expected. Also interesting to see that the statically linked interpreter is only 37Mb. Not too bad.

I will keep trying with Blas and Laplack I think, as I may need them for the libraries I'm trying to get working statically.

I am also going to have to try this with Scipy, which I read from the slither pages is harder, then after that I'll be trying to get it all working on Windows.

[patmarion]

After installing blas and lapack, did you rm the build directory completely before running run.sh again? The run.sh script might not remove anything, so maybe it did not cleanly re-configure numpy to find blas.

[jamesd256]

Yes I thought of that and removed build.

[patmarion]

Perhaps it has changed in newer versions of numpy, but in the included version numpy 1.6.2 I see this comment in numpy/core/setup.py: dotblas needs ATLAS, Fortran compiled blas will not be sufficient.

So try also installing libatlas-dev and then removing build and run run.sh one more time.

[jamesd256]

I tried as you suggested, and still have the same issue with the dotblas.so not being generated.

The numpy build process does seem to find a blas library:

blas_opt_info:
blas_mkl_info:
  libraries mkl,vml,guide not found in /home/jamesjdonnelly/NumpyBuiltinExample-master/build/install/lib
  libraries mkl,vml,guide not found in /usr/local/lib
  libraries mkl,vml,guide not found in /usr/lib
  NOT AVAILABLE
atlas_blas_threads_info:
Setting PTATLAS=ATLAS
  libraries ptf77blas,ptcblas,atlas not found in /home/jamesjdonnelly/NumpyBuiltinExample-master/build/install/lib
  libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib
  libraries ptf77blas,ptcblas,atlas not found in /usr/lib
  NOT AVAILABLE
atlas_blas_info:
  libraries f77blas,cblas,atlas not found in /home/jamesjdonnelly/NumpyBuiltinExample-master/build/install/lib
  libraries f77blas,cblas,atlas not found in /usr/local/lib
  libraries f77blas,cblas,atlas not found in /usr/lib
  NOT AVAILABLE
/home/jamesjdonnelly/NumpyBuiltinExample-master/build/numpy-1.6.2/numpy/distutils/system_info.py:1425: UserWarning: 
    Atlas (http://math-atlas.sourceforge.net/) libraries not found.
    Directories to search for the libraries can be specified in the
    numpy/distutils/site.cfg file (section [atlas]) or by setting
    the ATLAS environment variable.
  warnings.warn(AtlasNotFoundError.__doc__)
blas_info:
  libraries blas not found in /home/jamesjdonnelly/NumpyBuiltinExample-master/build/install/lib
  libraries blas not found in /usr/local/lib
  FOUND:
    libraries = ['blas']
    library_dirs = ['/usr/lib']
    language = f77
  FOUND:
    libraries = ['blas']
    library_dirs = ['/usr/lib']
    define_macros = [('NO_ATLAS_INFO', 1)]
    language = f77

[jamesd256]

The static atlas libraries are in libatlas-base-dev. I installed that, and now the _dotblas.so gets generated, but there follows some more undefined symbol stuff with the lapack stuff, which I am going to try to work through. First few:

/usr/lib/liblapack.a(cblas_cptaxpy.o): In function `cblas_caxpy':
(.text+0x11): undefined reference to `ATL_caxpy'
/usr/lib/liblapack.a(cblas_cptaxpy.o): In function `cblas_caxpy':
(.text+0x32): undefined reference to `ATL_caxpy'
/usr/lib/liblapack.a(cblas_cptaxpy.o): In function `cblas_caxpy'