Scitbx Tour: Setup and minimal framework add-on

Michael Hohn
Version $Id: scitbx-tour.txt,v 1.4 2005/03/02 01:01:26 rwgk Exp $


Borrowing the phenix infrastructure

Libraries

Among others, the SourceForge cctbx project at http://cctbx.sourceforge.net/current_cvs currently contains these modules:

boost
boost_adaptbx
A very small adaptor toolbox with platform-independent instructions for building the Boost.Python library.
libtbx
The build system common to all other modules. This is a very thin wrapper around the SCons software construction tool.
scitbx
Libraries for general scientific computing (i.e. libraries that are not specific to crystallographic applications): a family of high-level C++ array types, a fast Fourier transform library, and a C++ port of the popular L-BFGS quasi-Newton minimizer, all including Python bindings.
scons
The Python-based Make replacement.

For now, these five are the only ones needed.

Installation

Only source bundles have everything, so from http://cci.lbl.gov/cctbx_build get e.g.

wget http://cci.lbl.gov/cctbx_build/results/2005_01_22_0855/cctbx_python_24_bundle.selfx

if Python is not already installed, or it's too old (versions prior to 2.2). There is also a version w/o Python if the system's version is to be used.

Then, run

perl cctbx_python_24_bundle.selfx
in e.g.
sparx-sample
and give '0' cpus to prevent compilation.

This gives the directory

cctbx_sources

Now, follow this modified version of cctbxinstallscript.csh:

mkdir cctbx_build

cd cctbx_sources
\rm -fR ccp4io* clipper* cctbx/ PyCifRW iotbx mmtbx
cd ../cctbx_build
python ../cctbx_sources/libtbx/configure.py --build=release scitbx
source setpaths.csh
libtbx.scons include/scitbx/array_family/versa_algebra.h
libtbx.scons -j 4

to finish the library install. This latter part takes around 8 minutes on a 2.8 GHz P4.

The command

python ../cctbx_sources/libtbx/configure.py --build=release scitbx
prepares the cctbx_build/ directory and produces the toplevel "Makefile", cctbx_build/SConstruct.

Pre-use system setup

With the above build environment prepared, the shell needs to be set up before every new session; for this do

$ source cctbx_build/setpaths.csh 
to get the proper shell environment.

Extending with C++

The simplest thing is to add your own code into the framework. A complete component requires

  1. The C++ code.
  2. The boost.python wrapper (potentially in the same file).
  3. scons instructions for building.
The following sections contain cut-and-paste examples for a simple addition using existing facilities.

Setup

Prepare the module tree:

cd sparx-sample
source  cctbx_build/setpaths.csh 
cd cctbx_sources
mkdir voea
Minimal top-level SConscript:
$ cat >> ~/tmp/sparx-sample/cctbx_sources/voea/SConscript
SConscript("voea/SConscript")

And the directory structure:

mkdir voea/voea

Standalone sample, statically linked

The C++ library file

$ cat >> ~/tmp/sparx-sample/cctbx_sources/voea/voea/voea.cpp
#include "voea.h"
namespace voea {
        double 
        tiny_to_cpp(array::tiny<double, 2> t)
        {
                return (t[0] + t[1]);
        }
}
and its header file
$ cat >> ~/tmp/sparx-sample/cctbx_sources/voea/voea/voea.h
#ifndef VOEA_VOEA_H
#define VOEA_VOEA_H

#include <scitbx/array_family/tiny_types.h>

namespace voea {
namespace array = scitbx::af;

double 
tiny_to_cpp(array::tiny<double, 2> t);
}

#endif

The (trivial) C++ main() function.

$ cat >> ~/tmp/sparx-sample/cctbx_sources/voea/voea/test.cpp
#include "voea.h"
#include <iostream>

using namespace std;
namespace array = scitbx::af;

int
main(int argc, char **argv) {
        cout << voea::tiny_to_cpp( array::tiny<double, 2>(1.1, 2.2) ) << endl;
        return 0;
}

The module's dependencies, enumerated

$ cat >> ~/tmp/sparx-sample/cctbx_sources/voea/libtbx_config
{
  "modules_required_for_build": ["scitbx", "boost"],
  "modules_required_for_use": [],
}

The module's SConscript.

$ cat >> ~/tmp/sparx-sample/cctbx_sources/voea/voea/SConscript
import libtbx.load_env
# env_etc is NOT a scons Environment, but a data dictionary. 
Import("env_base", "env_etc")

env = env_base.Copy(
  CXXFLAGS=env_etc.cxxflags_base,
  LIBS=env_etc.libm,
  LIBPATH=["#lib"]
)

# For later or external references via $VOEA_DIST.
env_etc.voea_dist = libtbx.env.dist_path("voea")

env_etc.voea_common_includes = [
  env_etc.libtbx_include,
  env_etc.scitbx_include,
  env_etc.boost_include,
]

env_etc.include_registry.append(
  env = env,
  paths = env_etc.voea_common_includes)

# Standalone test, no library.
env.Program(source=["test.cpp", "voea.cpp"],
                        target="test-static")

Include the new module in the system via

pushd ../cctbx_build
python ../cctbx_sources/libtbx/configure.py voea

Now, the simple test can be built and run by

libtbx.scons voea/voea/test-static
./voea/voea/test-static 

A shared library version

Useful for larger projects, and needed later for the Python connection. The only addition is in the SConscript:

$ cat >> ~/tmp/sparx-sample/cctbx_sources/voea/voea/SConscript
# Shared Library
env.SharedLibrary(source=["voea.cpp"],
                                  target="#lib/voea")
# Shared library based test executable.
env_test = env.Copy()
env_test.Prepend(LIBS = ["voea"])
env_test.Program(source=["test.cpp"],
                                 target="test")

Compile and run via

libtbx.scons voea/voea/test
./voea/voea/test        

This executable now uses the library; on linux,

$ ldd ./voea/voea/test
shows
...
libvoea.so => ...cctbx_build/libtbx/libvoea.so (0x40017000)
...

Python connection

The additional C++ part:

$ cat >> ~/tmp/sparx-sample/cctbx_sources/voea/voea/voea_ext.cpp
/*
 * First include this !!!
 */
#include <scitbx/array_family/boost_python/flex_fwd.h> 
#include <scitbx/boost_python/container_conversions.h>
#include <scitbx/array_family/tiny_types.h>
#include <boost/python.hpp>
#include "voea.h"
BOOST_PYTHON_MODULE(voea_ext)
{
        using namespace boost::python;
        namespace array = scitbx::af;
        def("tiny_to_cpp", 
                (double (*)(array::tiny<double, 2>) ) voea::tiny_to_cpp );
}

The Python module additions:

$ cat >> ~/tmp/sparx-sample/cctbx_sources/voea/voea/__init__.py
#* C++ import redirections.
import scitbx.array_family.flex                 # register tuple mappings
import boost.python
ext = boost.python.import_ext("voea_ext")
from voea_ext import *

The SConscript additions:

$ cat >> ~/tmp/sparx-sample/cctbx_sources/voea/voea/SConscript
Import("env_scitbx_boost_python_ext")
# Borrow configuration.
env_bpl = env_scitbx_boost_python_ext.Copy()
env_bpl.Prepend(LIBS = ["voea"])
#
env_etc.include_registry.append(
  env=env_bpl,
  paths=env_etc.voea_common_includes)
#
env_bpl.SharedLibrary(
  target="#lib/voea_ext",
  source=["voea_ext.cpp"])
where #lib is a reference for the top-level build directory.

Compile everything

libtbx.scons .
and test in Python
$ python
import voea
print voea.tiny_to_cpp( [1,2] )
print voea.tiny_to_cpp( [1.1, 2] )

Some SConscript details

env_base
instance of scons' Environment holding basic settings
env_etc
instance of empty class shared across SConscript hierarchy (levels irrelevant, topological ordering matters) physical tree structure, dag logical structure. For utility only.
module dependencies
Always in MODULE/libtbx_config. e.g.
    {
      "modules_required_for_build": ["boost"],
      "modules_required_for_use": ["boost_adaptbx"],
    }

Here,

modules_required_for_build
include C++ building, linking, libraries.
modules_required_for_use
are the Python dependencies -- modules.

After changing libtbx_config, re-run

python ../cctbx_sources/libtbx/configure.py voea
in the build tree (the cwd becomes a build tree)
source setpaths.csh
libtbx.scons -j4

After rearrangements,

source unsetpaths.csh 

Unfinished notes

Python:

>>> from voea import trivial
>>> dir(trivial)
['__builtins__', '__doc__', '__file__', '__name__', 'boost', 'ext', 'test']
so the actual extension module is hidden as trivial.ext, and the Python wrapper can be modified. E.g.:
>>> foo = trivial.test
>>> trivial.test =          234123324
>>> trivial.test
234123324
>>> foo()
1

Even better,

trivial.ext.test 
is always available.

Useful:


The source directory:
echo $VOEA_DIST 

And so the test is

python $VOEA_DIST/voea/test.py 



This document was generated using AFT v5.095