[f2py] wrapping large Fortran code with f2py

[Vitaly Marin]

I am trying to generate a python interface to a large code, written in a
mix of Fortran and C, that has source in several sub-directories. I am
interested to wrap most of the top-level subroutines but
need everything else compiled. Since there are a all lot of Fortran
files it is not possible to pass all the information
to compile this code with f2py via the command line.

I have stumbled on a very similar posting at
http://osdir.com/ml/python.f2py.user/2004-01/msg00001.html

My question is how to complete step 4. My code contains multiple large
Makefiles to compile Fortran/C code (for each static library). I have
several static libraries.
How can I let f2py to generate the .so file or files? I would very
much appreciate the help.
More specific example would tremendously help me.

Thank you.
Vitaly

The copy of the posting found that I found

http://osdir.com/ml/python.f2py.user/2004-01/msg00001.html

is Below

1. Build the underlying Fortran libraries in full, as regular static
libs (libfoo.a).

2. Run f2py to generate a .pyf interface automatically. This is only a start.

3. Edit by hand the .pyf from step 2, and remove all references to any
function which is NOT needed from the python side. The python version
only exposes the functions which are specifically needed for the
'outside' world.

3.a [optional] Sometimes you may want to modify a bit how the
functions are exposed to python without messing with the fortran code,
and beyond what .pyf modifications can achieve. In this case, we make
a foo_.so and a foo.py which imports all from foo_.so, and makes the
appropriate changes. This is useful for improving docstrings, for
example, which can't currently be done via f2py.

4. Make a setup.py file which builds foo.so, including libfoo.a as a
library dependency for building foo.so.

This works extremely well for us here. I have a little tool written to
make the setup.py files as compact to write as possible (a few lines),
and a bunch of notes on this which are on the list archives. Email me
if you want the tool or can't find the old notes.

[Pearu Peterson]

Hi,

Post by Vitaly Marin
I am trying to generate a python interface to a large code, written in a
mix of Fortran and C, that has source in several sub-directories. I am
interested to wrap most of the top-level subroutines but
need everything else compiled. Since there are a all lot of Fortran
files it is not possible to pass all the information
to compile this code with f2py via the command line.
I have stumbled on a very similar posting at
http://osdir.com/ml/python.f2py.user/2004-01/msg00001.html

Btw, the statement in this message that it is not possible to build
extension module sources, is not true. One just should not use -c flag.
However, using setup.py in a Makefile is the recommended way.

Post by Vitaly Marin
My question is how to complete step 4. My code contains multiple large
Makefiles to compile Fortran/C code (for each static library). I have
several static libraries.
How can I let f2py to generate the .so file or files?

See

http://projects.scipy.org/scipy/numpy/wiki/DistutilsDoc

that explains how to use numpy.distutils for various tasks.
Examples of setup.py files can be found in (see in f2py_ext directory,
for instance)

http://projects.scipy.org/scipy/numpy/browser/trunk/numpy/distutils/tests/

See also setup.py files in scipy subpackages. For example,

http://projects.scipy.org/scipy/scipy/browser/trunk/scipy/fftpack/setup.py

shows how to build .a library from Fortran sources and how to link
it with an extension module.

If the Makefiles are too complex to convert the library building steps to
setup.py, then you can still use libraries built by Makefile in the setup.py
script: just specify the library names in `libraries` argument (if needed, also
the paths to libraries in `library_dirs` argument) and make sure that libraries
are built before executing setup.py command.

If you run into troubles in creating setup.py script from the above information,
just let us know and I can give more detailed help.

HTH,
Pearu

[Vitaly Marin]

Post by Pearu Peterson

Post by Vitaly Marin
I am trying to generate a python interface to a large code, written in a
mix of Fortran and C, that has source in several sub-directories. I am
interested to wrap most of the top-level subroutines but
need everything else compiled. Since there are a all lot of Fortran
files it is not possible to pass all the information
to compile this code with f2py via the command line.
I have stumbled on a very similar posting at
http://osdir.com/ml/python.f2py.user/2004-01/msg00001.html

Btw, the statement in this message that it is not possible to build
extension module sources, is not true. One just should not use -c flag.
However, using setup.py in a Makefile is the recommended way.

Post by Vitaly Marin
My question is how to complete step 4. My code contains multiple large
Makefiles to compile Fortran/C code (for each static library). I have
several static libraries.
How can I let f2py to generate the .so file or files?

See
http://projects.scipy.org/scipy/numpy/wiki/DistutilsDoc
that explains how to use numpy.distutils for various tasks.
Examples of setup.py files can be found in (see in f2py_ext directory,
for instance)
http://projects.scipy.org/scipy/numpy/browser/trunk/numpy/distutils/tests/
See also setup.py files in scipy subpackages. For example,
http://projects.scipy.org/scipy/scipy/browser/trunk/scipy/fftpack/setup.py
shows how to build .a library from Fortran sources and how to link
it with an extension module.
If the Makefiles are too complex to convert the library building steps to
setup.py, then you can still use libraries built by Makefile in the setup.py
script: just specify the library names in `libraries` argument (if needed, also
the paths to libraries in `library_dirs` argument) and make sure that libraries
are built before executing setup.py command.
If you run into troubles in creating setup.py script from the above information,
just let us know and I can give more detailed help.
HTH,
Pearu

Hi Pearu,

Yes. my Makefiles are complex. So I am trying to figure out how to
incorporate f2py with original Makefiles.

I was able to create signature files with ease for my Fortran modules
by writing a simple Perl script (I have hundreds of modules).

What I don't understand is if I have existing static libraries
(library.a in multiple directories, after running the original
Makefiles) how can I get python interface, since all the static
libraries are compiled?

Or what you mean is that I have to wrap up the main Fortran modules
and replace (/make changes to) main Makefile with setup.py (or update
the main Makefile) that would include the
static libraries that are needed for the main modules?

I would like to visualize this process so you would be able to correct
me on this. Below are the step I believe I have to go through

1) compile all static libraries using original Makefiles -----> for
each subdirectory
2) create signatures (.pyf) for all the top level modules but not
compile those with original Makefile (right?)
----I refinished (1) and (2) so far-----

3) create setup.py file that would replace the main Makefile and
include "libraries" parameter with the static libraries that I
completed in step 1.

Questions: is there a way for me to adjust may be Makefile instead,
say with f2py entry so it would call f2py to create shared library?

Normally, if we have to run f2py with hello.f90 I can run the
following two commands to accomplish this. But how can I incorporate
those static libraries and compile the top Fortran modules at the same
time?

My understanding is that I have to use f2py at the last step. The
default command is

(using simple hello.f90 example)

f2py --overwrite-signature hello.f90 -m hello -h hello.pyf
f2py -c --noarch --f90exec=/opt/intel_fcc_80/bin/ifort
--fcompiler=intel hello.pyf hello.f90

What should be my f2py command in case of including static libraries?
I really would like to understand this part?

Suppose, these are my static libraries
/mypath-to-static-lib/lib.a /mypath-to-static-lib/lib2.a .

Again, thank you in advance for your promptness

Vitaly Marin

[Pearu Peterson]

Hi Vitaly,

Post by Vitaly Marin
Yes. my Makefiles are complex. So I am trying to figure out how to
incorporate f2py with original Makefiles.
I was able to create signature files with ease for my Fortran modules
by writing a simple Perl script (I have hundreds of modules).
What I don't understand is if I have existing static libraries
(library.a in multiple directories, after running the original
Makefiles) how can I get python interface, since all the static
libraries are compiled?

f2py supports -l, -L, -I,.. arguments that have the same meaning as
for compilers/linkers. Also, you can add .o and .a files to f2py command
line and then they will be linked with the extension module.

Post by Vitaly Marin
Or what you mean is that I have to wrap up the main Fortran modules
and replace (/make changes to) main Makefile with setup.py (or update
the main Makefile) that would include the
static libraries that are needed for the main modules?

Yes, that can be an option.

Post by Vitaly Marin
I would like to visualize this process so you would be able to correct
me on this. Below are the step I believe I have to go through
1) compile all static libraries using original Makefiles -----> for
each subdirectory
2) create signatures (.pyf) for all the top level modules but not
compile those with original Makefile (right?)
----I refinished (1) and (2) so far-----

You can compile all Fortran sources. Note that in order to create
wrappers, f2py needs to know about Fortran function signatures.
The fastest way would be to read Fortran files and get signatures
from there. Another way is to create signature and f2py can
get function signatures from there, then f2py does not need
to read any Fortran files. In fact, when f2py command line contains
both .pyf file and a Fortran file, then f2py reads only the .pyf file
for signatures, it will complitely ignore the the Fortran files.

The fact that f2py can take all kinds of sources and build an
extension module from them, is a nice feature but is not the
main job for f2py. F2py job is to construct the extension module
sources. After that, user can use the sources to build extension
module as needed and without using f2py, in most cases this
approach is not used just because it is more convinient to let
f2py build the extension modules as it can handle lots of technical
details automatically and platform independent way
that otherwise one should do by hand.

Post by Vitaly Marin
3) create setup.py file that would replace the main Makefile and
include "libraries" parameter with the static libraries that I
completed in step 1.

setup.py needs not to replace the main Makefile, instead, the Makefile
can call the setup.py script to build extenson modules.

Post by Vitaly Marin
Questions: is there a way for me to adjust may be Makefile instead,
say with f2py entry so it would call f2py to create shared library?

Yes. Say, you have a m.pyf file with all necessary signatures and
a library mybuild/libfoo.a containing all Fortran symbols used
in the .pyf file. Then setup.py file would read, for example (untested):

def configuration(parent_package='',top_path=None):
from numpy.distutils.misc_util import Configuration
config = Configuration('',parent_package, top_path)
config.add_extension('m', sources=['m.pyf'], libraries=['foo'],
library_dirs=['mybuild'])
return config
if __name__=='__main__':
from numpy.distutils.core import setup
setup(configuration=configuration)

The main Makefile can call it as follows:

build_ext: <libfoo.a>
python setup.py build build_ext --inplace

Post by Vitaly Marin
Normally, if we have to run f2py with hello.f90 I can run the
following two commands to accomplish this. But how can I incorporate
those static libraries and compile the top Fortran modules at the same
time?

Hopefully the above example is an answer. Though it seems that you
could also skip creating setup.py file and call f2py from Makefile as
exemplified below.

Post by Vitaly Marin
My understanding is that I have to use f2py at the last step. The
default command is
(using simple hello.f90 example)
f2py --overwrite-signature hello.f90 -m hello -h hello.pyf
f2py -c --noarch --f90exec=/opt/intel_fcc_80/bin/ifort
--fcompiler=intel hello.pyf hello.f90
What should be my f2py command in case of including static libraries?
I really would like to understand this part?

You can add the static libraries to the f2py command line as mentioned above.

Post by Vitaly Marin
Suppose, these are my static libraries
/mypath-to-static-lib/lib.a /mypath-to-static-lib/lib2.a .

Then call f2py as follows:

f2py -c --noarch --f90exec=/opt/intel_fcc_80/bin/ifort
--fcompiler=intel hello.pyf hello.f90 /mypath-to-static-lib/lib.a
/mypath-to-static-lib/lib2.a

or

f2py -c --noarch --f90exec=/opt/intel_fcc_80/bin/ifort
--fcompiler=intel hello.pyf hello.f90 -L/mypath-to-static-lib/
-l<libraryname>

Here f2py will read hello.py for signaturs, constructs and compiles
Python extension module source, compiles hello.f90,
and finally links all .o and .a files together and creates the .so
file that can be imported to Python.

If you had compiled hello.f90 before, then you could use the following command:

f2py -c --noarch --f90exec=/opt/intel_fcc_80/bin/ifort
--fcompiler=intel hello.pyf hello.o /mypath-to-static-lib/lib.a
/mypath-to-static-lib/lib2.a

HTH,
Pearu

[Vitaly Marin]

Pearu, Thank you for your detailed information for f2py. I am going to
follow your advice on the
makefiles set up that will save me time and hopefully this wrapping
will work.

Vitaly

Post by Pearu Peterson

Post by Vitaly Marin
Yes. my Makefiles are complex. So I am trying to figure out how to
incorporate f2py with original Makefiles.
I was able to create signature files with ease for my Fortran modules
by writing a simple Perl script (I have hundreds of modules).
What I don't understand is if I have existing static libraries
(library.a in multiple directories, after running the original
Makefiles) how can I get python interface, since all the static
libraries are compiled?

f2py supports -l, -L, -I,.. arguments that have the same meaning as
for compilers/linkers. Also, you can add .o and .a files to f2py command
line and then they will be linked with the extension module.

Post by Vitaly Marin
Or what you mean is that I have to wrap up the main Fortran modules
and replace (/make changes to) main Makefile with setup.py (or update
the main Makefile) that would include the
static libraries that are needed for the main modules?

Yes, that can be an option.
HTH,
Pearu