+++++++++++++++++++++++++++++++++++++++++++++++++++++++
 Phil - Python-based hierarchical interchange language
+++++++++++++++++++++++++++++++++++++++++++++++++++++++

:Author: Ralf W. Grosse-Kunstleve
:Contact: RWGrosse-Kunstleve@lbl.gov

.. contents:: Sections
.. section-numbering::

**Links**

The primary home of this document is:
  http://cctbx.sourceforge.net/libtbx_phil.html

The source code examples below are available as one file here:
  http://cctbx.sourceforge.net/libtbx_phil_examples.py

``libtbx.phil`` is part of the cctbx open source libraries:
  http://cctbx.sourceforge.net/

===============
 Phil overview
===============

Phil (Python-based hierarchical interchange language) is a module
for the management of application parameters and, to some degree,
inputs. Many applications use command-line options as a user interface
(e.g. based on Python's optparse, a.k.a Optik). This approach works
well for small applications, but has it limitations for more complex
applications with a large set of parameters.

A simple Phil file as presented to the user may look like this::

  minimization.input {
    file_name = experiment.dat
    label = set2
  }
  minimization.output {
    model_file = final.mdl
    plot_file = None
  }
  minimization.parameters {
    method = *bfgs conjugate_gradient
    max_iterations = 10
  }

Phil is designed with a minimal syntax. An important goal is to
enable users to get started just by looking at defaults and examples,
without having to read documentation.

The Phil syntax has only two main elements, ``phil.definition`` (e.g.
``max_iterations = 10`` and ``phil.scope`` (e.g. ``minimization.input { }``).
To make this syntax as user-friendly as possible, strings
do not have to be quoted and, unlike Python, indentation is not
syntactically significant. E.g. this::

  minimization.input {
  file_name="experiment.dat"
  labels="set2"
  }

is equivalent to the corresponding definitions above.

Scopes can be nested recursively. The number of nesting levels is
limited only by Python's recursion limit (default 1000). To maximize
convenience, nested scopes can be defined in two equivalent ways.
For example::

  minimization {
    input {
    }
  }

is equivalent to::

  minimization.input {
  }

=============
Beyond syntax
=============

Phil is much more than just a parser for a very simple, user-friendly
syntax. Major Phil features are:

  - The concepts of `master files` and `user files`. The syntax
    for the two types of Phil files is identical, but the processed
    Phil files are used in different ways. I.e. the concepts exist
    only at the semantical level. The "look and feel" of the files is
    uniform.

  - Interpretation of command-line arguments as Phil definitions.

  - Merging of (multiple) Phil files and (multiple) Phil definitions
    derived from command-line arguments.

  - Automatic conversion of Phil files to Python objects
    which are essentially independent of the Phil system. I.e.
    core algorithms using Phil-derived parameter objects do not
    actually have to depend on Phil.

  - The reverse conversion of (potentially modified) Python
    objects back to Phil files. This could also be viewed as a Phil
    pretty printer.

  - Shell-like variable substitution using $var and ${var} syntax.

  - ``include`` syntax to merge Phil files at the parser level,
    or to import Phil objects from other Python scripts.

============
Master files
============

Master files are written by the software developer and include
"attributes" for each parameter, such as the type (integer,
floating-point, string, etc.) and support information for graphical
interfaces. For example::

  minimization.parameters
    .help = "Selection and tuning of minimization algorithm."
    .expert_level = 0
  {
    method = *bfgs conjugate_gradient
      .type = choice
    max_iterations = 10
      .type = int
      .input_size = 8
  }

The is the last part of the output of this command::

  libtbx.phil --show-some-attributes example.params

Run this command with ``--show-all-attributes`` to see the full set
of ``definition`` and ``scope`` attributes. This output tends to get
very long, but end-users don't have to be aware of this, and even
programmers only have to deal with the attributes they want to change.

==========
User files
==========

User files are typically generated by the application. For example::

  minimization.quick --show_defaults

will process its master file and show only the most
relevant parameters, classified by the software developer as
``.expert_level = 0`` (default). E.g. the ``minimization.parameters``
scope in the example above is not shown. The attributes are also
not shown. Therefore the output is much shorter compared to the
``libtbx.phil --show-some-attributes`` output above::

  minimization.parameters {
    method = *bfgs conjugate_gradient
    max_iterations = 10
  }

======================
Command-line arguments
======================

In theory the user could save and edit the generated parameter files.
However, in many practical situations this manual step can be
avoided. Phil is designed with the idea that the application inspects
all input files and uses the information found to fill in the blanks
automatically. This is not only convenient, but also eliminates
the possiblity of typing errors. In addition, the user can specify
parameters directly on the command line, and this information is also
use to fill in the blanks.

Command-line arguments that are not file names or options prefixed
with ``--`` (like ``--show_defaults`` above) should be given to Phil
for examination. E.g., this is a possible command::

  minimization.quick experiment.dat output.plot_file=plot.pdf

First the application should check if an argument is the name of a
file that can be opened. Assume this succeeds for the first argument,
so the processing of this argument is finished. Assume further that a
file with the name ``output.plot_file=plot.pdf`` does not exist. This
argument will therefore be interpreted with Phil. The next section
presents an example.

==============================
fetch: merging of Phil objects
==============================

The Phil parser converts master files, user files and command line
arguments to uniform Phil objects which can be merged to generate a
combined set of "working" parameters used in running the application.
We demonstrate this by way of a simple, self-contained Python script
with embedded Phil syntax::

  ## extract code begin: libtbx_phil_examples.py

  from libtbx.phil import parse

  master_phil = parse("""
    minimization.input {
      file_name = None
        .type = path
      label = None
        .type = str
    }
    """)

  user_phil = parse("""
    minimization.input {
      file_name = experiment.dat
    }
    """)

  command_line_phil = parse(
    "minimization.input.label=set2")

  working_phil = master_phil.fetch(
    sources=[user_phil, command_line_phil])
  working_phil.show()

  ## extract code end

``master_phil`` defines all available parameters including the
type information. ``user_phil`` overrides the default ``file_name``
assignment but leaves the ``labels`` undefined. These are defined
by a (fake) command-line argument. All inputs are merged via
``master_phil.fetch()``. ``working_phil.show()`` produces::

  minimization.input {
    file_name = experiment.dat
    label = set2
  }

Having to type in fully qualified parameter names (e.g.
``minimization.input.labels``) can be very inconvenient. Therefore
Phil includes support for matching parameter names of command-line
arguments as substrings to the parameter names in the master files::

  ## extract code begin: libtbx_phil_examples.py

  argument_interpreter = master_phil.command_line_argument_interpreter(
    home_scope="minimization")

  command_line_phil = argument_interpreter.process(
    arg="minimization.input.label=set2")

  ## extract code end

This works even if the user writes just ``label=set2`` or even
``put.lab=x1 x2``. The only requirement is that the substring leads
to a unique match in the master file. Otherwise Phil produces a helpful
error message. For example::

  argument_interpreter.process("a=set2")

leads to::

  Sorry: Ambiguous parameter definition: a = set2
  Best matches:
    minimization.input.file_name
    minimization.input.label

The user can cut-and-paste the desired parameter into the command
line for another trial to run the application.

=====================================================
extract: conversion of Phil objects to Python objects
=====================================================

The Phil parser produces objects that preserve most information
generated in the parsing process, such as line numbers and parameter
attributes. While this information is very useful for pretty printing
(e.g. to archive the working parameters) and the automatic generation
of graphical user interfaces, it is only a burden in the context of
core algorithms. Therefore Phil supports "extraction" of light-weight
Python objects from the Phil objects. Based on the example above,
this can be achieved with just one line::

  ## extract code begin: libtbx_phil_examples.py

  working_params = working_phil.extract()

  ## extract code end

We can now use the extracted objects in the context of Python::

  ## extract code begin: libtbx_phil_examples.py

  print working_params.minimization.input.file_name
  print working_params.minimization.input.label

  ## extract code end

Output::

  experiment.dat
  set2

``file_name`` and ``label`` are now a simple Python strings.

====================================================
format: conversion of Python objects to Phil objects
====================================================

Phil also supports the reverse conversion compared to the
previous section, from Python objects to Phil objects. For
example, to change the label::

  ## extract code begin: libtbx_phil_examples.py

  working_params.minimization.input.label = "set3"
  modified_phil = master_phil.format(python_object=working_params)
  modified_phil.show()

  ## extract code end

Output::

  minimization.input {
    file_name = "experiment.dat"
    label = "set3"
  }

We need to bring in ``master_phil`` again because all the meta
information was lost in the ``working_phil.extract()`` step that
produced ``working_params``. A type-specific converter is used to
produce a string for each Python object (see the Extending Phil
section below).

================
.multiple = True
================

Both ``phil.definition`` and ``phil.scope`` support the ``.multiple
= True`` attribute. For the sake of simplicity, in the following
"multiple definition" and "multiple scope" means a master definition or
scope with ``.multiple = True``. Please note the distinction between
this and multiple *values* given in a user file. For example, this
is a multiple definition in a master file::

  ## extract code begin: libtbx_phil_examples.py

  master_phil = parse("""
    minimization.input {
      file_name = None
        .type = path
        .multiple = True
    }
    """)

  ## extract code end

And these are multiple values for this definition in a user file::

  ## extract code begin: libtbx_phil_examples.py

  user_phil = parse("""
    minimization.input {
      file_name = experiment1.dat
      file_name = experiment2.dat
      file_name = experiment3.dat
    }
    """)

  ## extract code end

I.e. multiple values are simply specified by repeated definitions.
Without the ``.multiple = True`` in the master file, ``.fetch()``
retains only the *last* definition found in the master and all user
files or command-line arguments. ``.multiple = True`` directs Phil
to keep all values. ``.extract()`` then returns a list of all these
values converted to Python objects. For example, given the user
file above::

  ## extract code begin: libtbx_phil_examples.py

  working_params = master_phil.fetch(source=user_phil).extract()
  print working_params.minimization.input.file_name

  ## extract code end

will show this Python list::

  ['experiment1.dat', 'experiment2.dat', 'experiment3.dat']

Multiple scopes work similarly, for example::

  ## extract code begin: libtbx_phil_examples.py

  master_phil = parse("""
    minimization {
      input
        .multiple = True
      {
        file_name = None
          .type = path
        label = None
          .type = str
      }
    }
    """)

  ## extract code end

A corresponding user file may look this this::

  ## extract code begin: libtbx_phil_examples.py

  user_phil = parse("""
    minimization {
      input {
        file_name = experiment1.dat
        label = set2
      }
      input {
        file_name = experiment2.dat
        label = set1
      }
    }
    """)

  ## extract code end

The result of the usual fetch-extract sequence is::

  ## extract code begin: libtbx_phil_examples.py

  working_params = master_phil.fetch(source=user_phil).extract()
  for input in working_params.minimization.input:
    print input.file_name
    print input.label

  ## extract code end

Output::

  experiment1.dat
  set2
  experiment2.dat
  set1

Definitions and scopes may be nested with any combination of
``.multiple = False`` or ``.multiple = True``. For example, this
would be a plausible master file::

  ## extract code begin: libtbx_phil_examples.py

  master_phil = parse("""
    minimization {
      input
        .multiple = True
      {
        file_name = None
          .type = path
        label = None
          .type = str
          .multiple = True
      }
    }
    """)

  ## extract code end

This is a possible corresponding user file::

  ## extract code begin: libtbx_phil_examples.py

  user_phil = parse("""
    minimization {
      input {
        file_name = experiment1.dat
        label = set1
        label = set2
        label = set3
      }
      input {
        file_name = experiment2.dat
        label = set2
        label = set3
      }
    }
    """)

  ## extract code end

The fetch-extract sequence is the same as before::

  ## extract code begin: libtbx_phil_examples.py

  working_params = master_phil.fetch(source=user_phil).extract()
  for input in working_params.minimization.input:
    print input.file_name
    print input.label

  ## extract code end

but the output shows lists of strings for ``label`` instead of just
one Python string::

  experiment1.dat
  ['set1', 'set2', 'set3']
  experiment2.dat
  ['set2', 'set3']

===========================================================
fetch_diff: difference between master_phil and working_phil
===========================================================

The ``.fetch()`` method introduced above produces a complete copy
of the Phil master with all user definitions and scopes merged in.
If the Phil master is large, the output of ``working_phil.show()`` will
therefore also be large. It may be difficult to see which definitions
still have default values, and which definitions are changed.
To get just the difference between the master and the working
Phil objects, the ``.fetch_diff()`` method is available. For example::

  ## extract code begin: libtbx_phil_examples.py

  master_phil = parse("""
    minimization.parameters {
      method = *bfgs conjugate_gradient
        .type = choice
      max_iterations = 10
        .type = int
    }
    """)

  user_phil = parse("""
    minimization.parameters {
      method = bfgs *conjugate_gradient
    }
    """)

  working_phil = master_phil.fetch(source=user_phil)
  diff_phil = master_phil.fetch_diff(source=working_phil)
  diff_phil.show()

  ## extract code end

Output::

  minimization.parameters {
    method = bfgs *conjugate_gradient
  }

Here the minimization method was changed from ``bfgs``
to ``conjugate_gradient`` but the number of iterations is
unchanged. Therefore the latter does not appear in the output.
``.fetch_diff()`` is completely general and works for any combination
of definitions and scopes with ``.multiple = False`` or ``.multiple
= True``.

========
Includes
========

Phil also supports merging of files at the parsing level. For example::

  include file general.params

  minimization.parameters {
    include file specific.params
  }

Another option for building master files from a library of building
blocks is based on Python's import mechanism. For example::

  include file general.params

  minimization.parameters {
    include scope app.module.master_phil
  }

When encountering the ``include scope``, the Phil parser automatically
imports ``app.module`` (equivalent to ``import app.module`` in a Python
script). The ``master_phil`` object in the imported module must be
a pre-parsed Phil scope or a plain Phil string. The content of the
``master_phil`` scope is inserted into the scope of the ``include
scope`` statement.

``include`` directives enable hierarchical building of
master files without the need to copy-and-paste large fragments
explicitly. Duplication appears only in automatically generated user
files. I.e. the programmer is well served because a system of master
files can be kept free of large-scale redundancies that are difficult
to maintain. At the same time the end user is well served because
the indirections are resolved automatically and all parameters are
presented in one uniform view.

=====================
Variable substitution
=====================

Phil supports variable substitution using $var and $(var)
syntax. A few examples say more than many words::

  ## extract code begin: libtbx_phil_examples.py

  var_phil = parse("""
    root_name = peak
    file_name = $root_name.mtz
    full_path = $HOME/$file_name
    related_file_name = $(root_name)_data.mtz
    message = "Reading $file_name"
    as_is = ' $file_name '
    """)
  var_phil.fetch(source=var_phil).show()

  ## extract code end

Output::

  root_name = peak
  file_name = "peak.mtz"
  full_path = "/net/cci/rwgk/peak.mtz"
  related_file_name = "peak_data.mtz"
  message = "Reading peak.mtz"
  as_is = ' $file_name '

Note that the variable substitution does not happen during parsing.
The output of ``params.show`` is identical to the input. In the
example above, variables are substituted by the ``.fetch()`` method
that we introduced earlier to merge user files given a master file.

==============
Extending Phil
==============

Phil comes with a number of predefined converters used by ``.extract()``
and ``.format()`` to convert to and from pure Python objects. These
are::

  .type =
    words     retains the "words" produced by the parser
    strings   list of Python strings (also used for .type = None)
    str       combines all words into one string
    path      path name (same as str_converters)
    key       database key (same as str_converters)
    bool      Python bool
    int       Python int
    float     Python float
    choice    string selected from a pre-defined list

It is possible to extend Phil with user-defined converters.
For example::

  ## extract code begin: libtbx_phil_examples.py

  import libtbx.phil
  from libtbx.phil import tokenizer

  class upper_converters:

    phil_type = "upper"

    def __str__(self): return self.phil_type

    def from_words(self, words, master):
      s = libtbx.phil.str_from_words(words=words)
      if (s is None): return None
      return s.upper()

    def as_words(self, python_object, master):
      if (python_object is None):
        return [tokenizer.word(value="None")]
      return [tokenizer.word(value=python_object.upper())]

  converter_registry = libtbx.phil.extended_converter_registry(
    additional_converters=[upper_converters])

  ## extract code end

The extended ``converter_registry`` is passed as an additional
argument to Phil's ``parse`` function::

  ## extract code begin: libtbx_phil_examples.py

  master_phil = parse("""
    value = None
      .type = upper
    """,
      converter_registry=converter_registry)
  user_phil = parse("value = extracted")
  working_params = master_phil.fetch(source=user_phil).extract()
  print working_params.value

  ## extract code end

The ``print`` statement at the end writes "EXTRACTED". It also goes
the other way, starting with a lower-case Python value::

  ## extract code begin: libtbx_phil_examples.py

  working_params.value = "formatted"
  working_phil = master_phil.format(python_object=working_params)
  working_phil.show()

  ## extract code end

The output of the ``.show()`` call is "value = FORMATTED".

Arbitrary new types can be added to Phil by defining similar
converters. If desired, the pre-defined converters for the basic
types can even be replaced. All converters have to have ``__str__()``,
``from_words()`` and ``as_words()`` methods. More complex converters
may optionally have a non-trivial ``__init__()`` method (an example
is the ``choice_converters`` class in ``libtbx/phil/__init__.py``).

Additional domain-specific converters are best defined in a separate
module, along with a corresponding parse() function using the
extended converter registry as the default. See, for example,
``iotbx/phil.py`` in the same ``cctbx`` project that also hosts
``libtbx``.

=======
Details
=======

.type = ints and .type = floats
-------------------------------

The built-in ``ints`` and ``floats`` converters handle lists of
integer and floating point numbers, respectively. For example::

  ## extract code begin: libtbx_phil_examples.py

  master_phil = parse("""
    random_integers = None
      .type = ints
    euler_angles = None
      .type = floats(size=3)
    unit_cell_parameters = None
      .type = floats(size_min=1, size_max=6)
    rotation_part = None
      .type = ints(size=9, value_min=-1, value_max=1)
    """)

  user_phil = parse("""
    random_integers = 3 18 5
    euler_angles = 10 -20 30
    unit_cell_parameters = 10,20,30
    rotation_part = "1,0,0;0,-1,0;0,0,-1"
    """)

  working_phil = master_phil.fetch(source=user_phil)
  working_phil.show()
  print
  working_params = working_phil.extract()
  print working_params.random_integers
  print working_params.euler_angles
  print working_params.unit_cell_parameters
  print working_params.rotation_part
  print
  working_phil = master_phil.format(python_object=working_params)
  working_phil.show()

  ## extract code end

Output::

  random_integers = 3 18 5
  euler_angles = 10 -20 30
  unit_cell_parameters = 10,20,30
  rotation_part = "1,0,0;0,-1,0;0,0,-1"

  [3, 18, 5]
  [10.0, -20.0, 30.0]
  [10.0, 20.0, 30.0]
  [1, 0, 0, 0, -1, 0, 0, 0, -1]

  random_integers = 3 18 5
  euler_angles = 10 -20 30
  unit_cell_parameters = 10 20 30
  rotation_part = 1 0 0 0 -1 0 0 0 -1

The list of ``random_integers`` can have arbitrary size and arbitrary
values.

For ``euler_angles``, exactly three values must be given.

For ``unit_cell_parameters``, one to six values are acceptable.

The list of values for ``rotation_part`` must have nine integer
elements, with values {-1,0,1}.

All keywords are optional and can be used in any combination, except if
``size`` is given, ``size_min`` and ``size_max`` cannot also be given.

Lists of values can optionally use commas or semicolons as separators
between values. In this context, both characters are equivalent to
a white-space. ``.format()`` always uses spaces as separators, i.e.
commas and semicolons are not preserved in an ``.extract()``-``.format()``
cycle. (Note that lists using semicolons as separators must be quoted;
see the "Semicolon syntax" section below.)

.type = choice
--------------

The built-in ``choice`` converters support single and multi choices.
Here are two examples, a single choice ``gender`` and a multi choice
``favorite_sweets``::

  ## extract code begin: libtbx_phil_examples.py

  master_phil = parse("""
    gender = male female
      .type = choice
    favorite_sweets = ice_cream chocolate candy_cane cookies
      .type = choice(multi=True)
    """)

  jims_choices = parse("""
    gender = *male female
    favorite_sweets = *ice_cream chocolate candy_cane *cookies
    """)

  jims_phil = master_phil.fetch(source=jims_choices)
  jims_phil.show()
  jims_params = jims_phil.extract()
  print jims_params.gender, jims_params.favorite_sweets

  ## extract code end

Selected items are marked with a star ``*``. The ``.extract()``
method returns either a string with the selected value (single choice)
or a list of strings with all selected values (multi choice). The
output of the example is::

  gender = *male female
  favorite_sweets = *ice_cream chocolate candy_cane *cookies
  male ['ice_cream', 'cookies']

To maximize convenience, especially for choices specified via the
command-line, the ``*`` is optional if only one value is given.
For example, the following two definitions are equivalent::

  gender = female
  gender = male *female

If the ``.optional`` attribute is not defined, it defaults to ``True``
and this is possible::

  ## extract code begin: libtbx_phil_examples.py

  ignorant_choices = parse("""
    gender = male female
    favorite_sweets = ice_cream chocolate candy_cane cookies
    """)

  ignorant_params = master_phil.fetch(source=ignorant_choices).extract()
  print ignorant_params.gender, ignorant_params.favorite_sweets

  ## extract code end

Output::

  None []

In this case the application has to deal with the ``None`` and
the empty list. If ``.optional = False``, ``.extract()`` will lead
to informative error messages. The application will never receive
``None`` or an empty list.

If a value in the user file is not a possible choice, ``.extract()``
leads to an error message listing all possible choices, for example::

  Sorry: Not a possible choice for favorite_sweets: icecream
    Possible choices are:
      ice_cream
      chocolate
      candy_cane
      cookies

This message is designed to aid users in recovering from mis-spelled
choices typed in at the command-line. Command-line choices are
further supported by this syntax::

  ## extract code begin: libtbx_phil_examples.py

  greedy_choices = parse("""
    favorite_sweets=ice_cream+chocolate+cookies
    """)

  greedy_params = master_phil.fetch(source=greedy_choices).extract()
  print greedy_params.favorite_sweets

  ## extract code end

Ouput::

  ['ice_cream', 'chocolate', 'cookies']

Finally, if the ``.optional`` attribute is not specified or ``True``,
``None`` can be assigned::

  ## extract code begin: libtbx_phil_examples.py

  no_thanks_choices = parse("""
    favorite_sweets=None
    """)

  no_thanks_params = master_phil.fetch(source=no_thanks_choices).extract()
  print no_thanks_params.favorite_sweets

  ## extract code end

Output::

  []

scope_extract
-------------

The result of ``scope.extract()`` is a ``scope_extract`` instance
with attributes corresponding to the embedded definitions and
sub-scopes. For example::

  ## extract code begin: libtbx_phil_examples.py

  master_phil = parse("""
    minimization.input {
      file_name = None
        .type = path
    }
    minimization.parameters {
      max_iterations = 10
        .type = int
    }
    """)

  user_phil = parse("""
    minimization.input.file_name = experiment.dat
    minimization.parameters.max_iterations = 5
    """)

  working_params = master_phil.fetch(source=user_phil).extract()
  print working_params
  print working_params.minimization.input.file_name
  print working_params.minimization.parameters.max_iterations

  ## extract code end

Output::

  <libtbx.phil.scope_extract object at 0x2ad50bae7550>
  experiment.dat
  5

This just repeats what was shown several times before, but
``scope_extract`` includes a few additional, special features that are
worth knowing. The first special feature is the ``.__phil_path__()``
method::

  ## extract code begin: libtbx_phil_examples.py

  print working_params.minimization.input.__phil_path__()
  print working_params.minimization.parameters.__phil_path__()

  ## extract code end

Output::

  minimization.input
  minimization.parameters

This feature is most useful for formatting informative error messages
without having to hard-wire the fully-qualified parameter names. Use
``.__phil_path__()`` to ensure that the names are automatically
correct even if the master file is changed in major ways. Note that the
``.__phil_path__()`` method is available only for extracted scopes,
not for extracted definitions since it would be very cumbersome to
implement. However, the fully-qualified name of a definition can
be obtained via ``.__phil_path__(object_name="max_iterations")``;
usually the ``object_name`` is readily available in the contexts in
which the fully-qualified name is needed. There is also
``.__phil_path_and_value__(object_name)`` which returns a 2-tuple
of the fully-qualified path and the extracted value, ready to
be used for formatting error messages.

The next important feature is a safety guard: assignment to a
non-existing attribute leads to an exception. For example,
if the attribute is mis-spelled::

  working_params.minimization.input.filename = "other.dat"

Result::

  AttributeError: Assignment to non-existing attribute "minimization.input.filename"
    Please correct the attribute name, or to create
    a new attribute use: obj.__inject__(name, value)

In addition to trivial spelling errors, the safety guard traps
overlooked dependencies related to changes in the master file.

In some (unusual) situations it may be useful to attach attributes to
an extracted scope that have no correspondence in the master file.
Use the ``.__inject__(name, value)`` method for this purpose to
by-pass the safety-guard. As a side-effect of this design, injected
attributes are easily pin-pointed in the source code (simply search
for ``__inject__``), which can be a big help in maintaining a large
code base.

Multiple definitions and scopes
-------------------------------

All Phil attributes of multiple definitions or scopes are determined
by the first occurrence in the master file. All following instances in
the master file are defaults. Any instances in user files (merged via
``.fetch()``) are added to the default instances in the master file.
For example::

  ## extract code begin: libtbx_phil_examples.py

  master_phil = parse("""
    plot
      .multiple = True
    {
      style = line bar pie_chart
        .type=choice
      title = None
        .type = str
    }
    plot {
      style = line
      title = Line plot (default in master)
    }
    """)

  user_phil = parse("""
    plot {
      style = bar
      title = Bar plot (provided by user)
    }
    """)

  working_phil = master_phil.fetch(source=user_phil)
  working_phil.show()

  ## extract code end

Output::

  plot {
    style = *line bar pie_chart
    title = Line plot (default in master)
  }
  plot {
    style = line *bar pie_chart
    title = Bar plot (provided by user)
  }

``.extract()`` will produce a list with two elements::

  ## extract code begin: libtbx_phil_examples.py

  working_params = working_phil.extract()
  print working_params.plot

  ## extract code end

Output::

  [<libtbx.phil.scope_extract object at 0x2b1ccb5b1910>,
   <libtbx.phil.scope_extract object at 0x2b1ccb5b1c10>]

Note that the first (i.e. master) occurrence of the scope is not
extracted. In practice this is usually the desired behavior, but it
can be changed by setting the ``plot`` scope attribute ``.optional
= False``. For example::

  ## extract code begin: libtbx_phil_examples.py

  master_phil = parse("""
    plot
      .multiple = True
      .optional = False
    {
      style = line bar pie_chart
        .type=choice
      title = None
        .type = str
    }
    plot {
      style = line
      title = Line plot (default in master)
    }
    """)

  ## extract code end

With the ``user_phil`` as before, ``.show()`` and ``.extract()`` now
produce three entries each::

  ## extract code begin: libtbx_phil_examples.py

  working_phil = master_phil.fetch(source=user_phil)
  working_phil.show()
  print working_phil.extract().plot

  ## extract code end

Output::

  plot {
    style = line bar pie_chart
    title = None
  }
  plot {
    style = *line bar pie_chart
    title = Line plot (default in master)
  }
  plot {
    style = line *bar pie_chart
    title = Bar plot (provided by user)
  }
  [<libtbx.phil.scope_extract object at 0x2af4c307bcd0>,
   <libtbx.phil.scope_extract object at 0x2af4c307bd50>,
   <libtbx.phil.scope_extract object at 0x2af4c307be10>]

With ``.optional = True``, the master of a multiple definition or
scope is *never* extracted. With ``.optional = False``, it is *always*
extracted, and always first in the list.

The "always first in the list" rule for multiple master objects is
special. Other instances of multiple scopes are shown and extracted
in the order in which they appear in the master file and the merged
user file(s), with all *exact* duplicates removed. If duplicates are
detected, the earlier copy is removed, unless it is the master.

These rules are designed to produce easily predictable results in
situations where multiple Phil files are merged (via ``.fetch()``),
including complete copies of the master file.

fetch option: track_unused_definitions
--------------------------------------

The default behavior of ``.fetch()`` is to simply ignore user
definitions that don't match anything in the master file. It it is
possible to request a complete list of all user definitions ignored by
``.fetch()``. For example::

  ## extract code begin: libtbx_phil_examples.py

  master_phil = parse("""
    input {
      file_name = None
        .type = path
    }
    """)

  user_phil = parse("""
    input {
      file_name = experiment.dat
      label = set1
      lable = set2
    }
    """)

  working_phil, unused = master_phil.fetch(
    source=user_phil, track_unused_definitions=True)
  working_phil.show()
  for object_locator in unused:
    print "unused:", object_locator

  ## extract code end

Output::

  input {
    file_name = experiment.dat
  }
  unused: input.label (input line 4)
  unused: input.lable (input line 5)

To catch spelling errors, or to alert users to changes in the master
file, it is good practice to set ``track_unused_definitions=True``
and to show warnings or errors.

Semicolon syntax
----------------

In all the examples above, line breaks act as syntactical elements
delineating the end of definitions. This is most obvious, but for
convenience, Phil also supports using the semicolon ``;`` instead.
For example::

  ## extract code begin: libtbx_phil_examples.py

  phil_scope = parse("""
     quick .multiple=true;.optional=false{and=very;.type=str;dirty=use only on command-lines, please!;.type=str}
     """)

  phil_scope.show(attributes_level=2)

  ## extract code end

Clearly, the output looks much nicer::

  quick
    .optional = False
    .multiple = True
  {
    and = very
      .type = str
    dirty = use only on command-lines, please!
      .type = str
  }

Master files generally shouldn't make use of the semicolon syntax, even
though it is possible. In user files it is more acceptable, but the main
purpose is to support passing parameters from the command line.

Note that the Phil output methods (``.show()``, ``.as_str()``) never
make use of the semicolon syntax.

Phil comments
-------------

Phil supports two types of comments:

  - Simple one-line comments starting with a hash ``#``.
    All following characters through the end of the line are ignored.

  - Syntax-aware comments starting with an exclamation mark ``!``.

The exclamation mark can be used to easily comment out entire
syntactical constructs, for example a complete scope including all
attributes::

  ## extract code begin: libtbx_phil_examples.py

  master_phil = parse("""
    !input {
      file_name = None
        .type = path
        .multiple = True
    }
    """)
  master_phil.show()

  ## extract code end

Output::

  !input {
    file_name = None
  }

As is evident from the output, Phil keeps the content "in mind",
but the scope is not actually used by ``.fetch()``::

  ## extract code begin: libtbx_phil_examples.py

  user_phil = parse("""
    input.file_name = experiment.dat
    """)
  print len(master_phil.fetch(source=user_phil).as_str())

  ## extract code end

Output::

  0

I.e. the ``.fetch()`` method ignored the user definition because the
corresponding master is commented out.

Quotes and backslash
--------------------

Similar to Python, Phil supports single quotes, double quotes,
and triple quotes (three single or three double quotes). Unlike
Python, quotes can often be omitted, and single quotes and double
quotes have different semantics, similar to that of Unix shells:
``$`` variables are expanded if embedded in double quotes, but not
if embedded in single quotes. See the variable substitution section
above for examples.

The backslash can be used in the usual way (Python, Unix shells) to
"escape" line breaks, quotes, and a second backslash.

For convenience, a line starting with quotes is automatically treated
as a continuation of a definition on the previous line(s). The trailing
backslash on the previous line may be omitted.

The exact rules for quoting and backslash escapes are intricate.
A significant effort was made to mimic the familiar behavior of Python
and Unix shells where possible, but nested constructs of quotes and
backslashes are still prone to cause surprises. In unusual situations,
probably the fastest method to obtain the desired result is trial
and error (as opposed to studying the intricate rules).

scope.show(attributes_level=3)
------------------------------

In this document, the ``scope.show()`` method is used extensively
in the examples. With the defaults for the method parameters, it
only shows the Phil scope or definition names and and associated
values. It is also possible to include some or all Phil scope or
definition attributes in the ``.show()`` output, as directed by the
``attributes_level`` parameter::

  attributes_level=0: shows only names and values
                   1: also shows the .help attribute
                   2: shows all attributes which are not None
                   3: shows all attributes

``scope.show(attributes_level=2)`` can be used to pretty-print
master files without any loss of information. ``attributes_level=3``
is useful to obtain a full listing of all available attributes,
but all information is preserved with the usually much less verbose
``attributes_level=2``. This is illustrated by the following example::

  ## extract code begin: libtbx_phil_examples.py

  master_phil = parse("""
    minimization {
      input
        .help = "File names and data labels."
        .multiple = True
      {
        file_name = None
          .type = path
        label = None
          .help = "A unique substring of the data label is sufficient."
          .type = str
      }
    }
    """)

  for attributes_level in xrange(4):
    master_phil.show(attributes_level=attributes_level)

  ## extract code end

Output with ``attributes_level=0`` (the default)::

  minimization {
    input {
      file_name = None
      label = None
    }
  }

Output with ``attributes_level=1``::

  minimization {
    input
      .help = "File names and data labels."
    {
      file_name = None
      label = None
        .help = "A unique substring of the data label is sufficient."
    }
  }

Output with ``attributes_level=2``::

  minimization {
    input
      .help = "File names and data labels."
      .multiple = True
    {
      file_name = None
        .type = path
      label = None
        .help = "A unique substring of the data label is sufficient."
        .type = str
    }
  }

Output with ``attributes_level=3``::

  minimization
    .style = None
    .help = None
    .caption = None
    .short_caption = None
    .optional = None
    .call = None
    .multiple = None
    .sequential_format = None
    .disable_add = None
    .disable_delete = None
    .expert_level = None
  {
    input
      .style = None
      .help = "File names and data labels."
      .caption = None
      .short_caption = None
      .optional = None
      .call = None
      .multiple = True
      .sequential_format = None
      .disable_add = None
      .disable_delete = None
      .expert_level = None
    {
      file_name = None
        .help = None
        .caption = None
        .short_caption = None
        .optional = None
        .type = path
        .multiple = None
        .input_size = None
        .expert_level = None
      label = None
        .help = "A unique substring of the data label is sufficient."
        .caption = None
        .short_caption = None
        .optional = None
        .type = str
        .multiple = None
        .input_size = None
        .expert_level = None
    }
  }