1.1 --- /dev/null Thu Jan 01 00:00:00 1970 +0000
1.2 +++ b/python/mach/README.rst Wed Dec 31 06:09:35 2014 +0100
1.3 @@ -0,0 +1,328 @@
1.4 +====
1.5 +mach
1.6 +====
1.7 +
1.8 +Mach (German for *do*) is a generic command dispatcher for the command
1.9 +line.
1.10 +
1.11 +To use mach, you install the mach core (a Python package), create an
1.12 +executable *driver* script (named whatever you want), and write mach
1.13 +commands. When the *driver* is executed, mach dispatches to the
1.14 +requested command handler automatically.
1.15 +
1.16 +Features
1.17 +========
1.18 +
1.19 +On a high level, mach is similar to using argparse with subparsers (for
1.20 +command handling). When you dig deeper, mach offers a number of
1.21 +additional features:
1.22 +
1.23 +Distributed command definitions
1.24 + With optparse/argparse, you have to define your commands on a central
1.25 + parser instance. With mach, you annotate your command methods with
1.26 + decorators and mach finds and dispatches to them automatically.
1.27 +
1.28 +Command categories
1.29 + Mach commands can be grouped into categories when displayed in help.
1.30 + This is currently not possible with argparse.
1.31 +
1.32 +Logging management
1.33 + Mach provides a facility for logging (both classical text and
1.34 + structured) that is available to any command handler.
1.35 +
1.36 +Settings files
1.37 + Mach provides a facility for reading settings from an ini-like file
1.38 + format.
1.39 +
1.40 +Components
1.41 +==========
1.42 +
1.43 +Mach is conceptually composed of the following components:
1.44 +
1.45 +core
1.46 + The mach core is the core code powering mach. This is a Python package
1.47 + that contains all the business logic that makes mach work. The mach
1.48 + core is common to all mach deployments.
1.49 +
1.50 +commands
1.51 + These are what mach dispatches to. Commands are simply Python methods
1.52 + registered as command names. The set of commands is unique to the
1.53 + environment mach is deployed in.
1.54 +
1.55 +driver
1.56 + The *driver* is the entry-point to mach. It is simply an executable
1.57 + script that loads the mach core, tells it where commands can be found,
1.58 + then asks the mach core to handle the current request. The driver is
1.59 + unique to the deployed environment. But, it's usually based on an
1.60 + example from this source tree.
1.61 +
1.62 +Project State
1.63 +=============
1.64 +
1.65 +mach was originally written as a command dispatching framework to aid
1.66 +Firefox development. While the code is mostly generic, there are still
1.67 +some pieces that closely tie it to Mozilla/Firefox. The goal is for
1.68 +these to eventually be removed and replaced with generic features so
1.69 +mach is suitable for anybody to use. Until then, mach may not be the
1.70 +best fit for you.
1.71 +
1.72 +Implementing Commands
1.73 +---------------------
1.74 +
1.75 +Mach commands are defined via Python decorators.
1.76 +
1.77 +All the relevant decorators are defined in the *mach.decorators* module.
1.78 +The important decorators are as follows:
1.79 +
1.80 +CommandProvider
1.81 + A class decorator that denotes that a class contains mach
1.82 + commands. The decorator takes no arguments.
1.83 +
1.84 +Command
1.85 + A method decorator that denotes that the method should be called when
1.86 + the specified command is requested. The decorator takes a command name
1.87 + as its first argument and a number of additional arguments to
1.88 + configure the behavior of the command.
1.89 +
1.90 +CommandArgument
1.91 + A method decorator that defines an argument to the command. Its
1.92 + arguments are essentially proxied to ArgumentParser.add_argument()
1.93 +
1.94 +Classes with the *@CommandProvider* decorator *must* have an *__init__*
1.95 +method that accepts 1 or 2 arguments. If it accepts 2 arguments, the
1.96 +2nd argument will be a *MachCommandContext* instance. This is just a named
1.97 +tuple containing references to objects provided by the mach driver.
1.98 +
1.99 +Here is a complete example::
1.100 +
1.101 + from mach.decorators import (
1.102 + CommandArgument,
1.103 + CommandProvider,
1.104 + Command,
1.105 + )
1.106 +
1.107 + @CommandProvider
1.108 + class MyClass(object):
1.109 + @Command('doit', help='Do ALL OF THE THINGS.')
1.110 + @CommandArgument('--force', '-f', action='store_true',
1.111 + help='Force doing it.')
1.112 + def doit(self, force=False):
1.113 + # Do stuff here.
1.114 +
1.115 +When the module is loaded, the decorators tell mach about all handlers.
1.116 +When mach runs, it takes the assembled metadata from these handlers and
1.117 +hooks it up to the command line driver. Under the hood, arguments passed
1.118 +to the decorators are being used to help mach parse command arguments,
1.119 +formulate arguments to the methods, etc. See the documentation in the
1.120 +*mach.base* module for more.
1.121 +
1.122 +The Python modules defining mach commands do not need to live inside the
1.123 +main mach source tree.
1.124 +
1.125 +Conditionally Filtering Commands
1.126 +--------------------------------
1.127 +
1.128 +Sometimes it might only make sense to run a command given a certain
1.129 +context. For example, running tests only makes sense if the product
1.130 +they are testing has been built, and said build is available. To make
1.131 +sure a command is only runnable from within a correct context, you can
1.132 +define a series of conditions on the *Command* decorator.
1.133 +
1.134 +A condition is simply a function that takes an instance of the
1.135 +*CommandProvider* class as an argument, and returns True or False. If
1.136 +any of the conditions defined on a command return False, the command
1.137 +will not be runnable. The doc string of a condition function is used in
1.138 +error messages, to explain why the command cannot currently be run.
1.139 +
1.140 +Here is an example:
1.141 +
1.142 + from mach.decorators import (
1.143 + CommandProvider,
1.144 + Command,
1.145 + )
1.146 +
1.147 + def build_available(cls):
1.148 + """The build needs to be available."""
1.149 + return cls.build_path is not None
1.150 +
1.151 + @CommandProvider
1.152 + class MyClass(MachCommandBase):
1.153 + def __init__(self, build_path=None):
1.154 + self.build_path = build_path
1.155 +
1.156 + @Command('run_tests', conditions=[build_available])
1.157 + def run_tests(self):
1.158 + # Do stuff here.
1.159 +
1.160 +It is important to make sure that any state needed by the condition is
1.161 +available to instances of the command provider.
1.162 +
1.163 +By default all commands without any conditions applied will be runnable,
1.164 +but it is possible to change this behaviour by setting *require_conditions*
1.165 +to True:
1.166 +
1.167 + m = mach.main.Mach()
1.168 + m.require_conditions = True
1.169 +
1.170 +Minimizing Code in Commands
1.171 +---------------------------
1.172 +
1.173 +Mach command modules, classes, and methods work best when they are
1.174 +minimal dispatchers. The reason is import bloat. Currently, the mach
1.175 +core needs to import every Python file potentially containing mach
1.176 +commands for every command invocation. If you have dozens of commands or
1.177 +commands in modules that import a lot of Python code, these imports
1.178 +could slow mach down and waste memory.
1.179 +
1.180 +It is thus recommended that mach modules, classes, and methods do as
1.181 +little work as possible. Ideally the module should only import from
1.182 +the *mach* package. If you need external modules, you should import them
1.183 +from within the command method.
1.184 +
1.185 +To keep code size small, the body of a command method should be limited
1.186 +to:
1.187 +
1.188 +1. Obtaining user input (parsing arguments, prompting, etc)
1.189 +2. Calling into some other Python package
1.190 +3. Formatting output
1.191 +
1.192 +Of course, these recommendations can be ignored if you want to risk
1.193 +slower performance.
1.194 +
1.195 +In the future, the mach driver may cache the dispatching information or
1.196 +have it intelligently loaded to facilitate lazy loading.
1.197 +
1.198 +Logging
1.199 +=======
1.200 +
1.201 +Mach configures a built-in logging facility so commands can easily log
1.202 +data.
1.203 +
1.204 +What sets the logging facility apart from most loggers you've seen is
1.205 +that it encourages structured logging. Instead of conventional logging
1.206 +where simple strings are logged, the internal logging mechanism logs all
1.207 +events with the following pieces of information:
1.208 +
1.209 +* A string *action*
1.210 +* A dict of log message fields
1.211 +* A formatting string
1.212 +
1.213 +Essentially, instead of assembling a human-readable string at
1.214 +logging-time, you create an object holding all the pieces of data that
1.215 +will constitute your logged event. For each unique type of logged event,
1.216 +you assign an *action* name.
1.217 +
1.218 +Depending on how logging is configured, your logged event could get
1.219 +written a couple of different ways.
1.220 +
1.221 +JSON Logging
1.222 +------------
1.223 +
1.224 +Where machines are the intended target of the logging data, a JSON
1.225 +logger is configured. The JSON logger assembles an array consisting of
1.226 +the following elements:
1.227 +
1.228 +* Decimal wall clock time in seconds since UNIX epoch
1.229 +* String *action* of message
1.230 +* Object with structured message data
1.231 +
1.232 +The JSON-serialized array is written to a configured file handle.
1.233 +Consumers of this logging stream can just perform a readline() then feed
1.234 +that into a JSON deserializer to reconstruct the original logged
1.235 +message. They can key off the *action* element to determine how to
1.236 +process individual events. There is no need to invent a parser.
1.237 +Convenient, isn't it?
1.238 +
1.239 +Logging for Humans
1.240 +------------------
1.241 +
1.242 +Where humans are the intended consumer of a log message, the structured
1.243 +log message are converted to more human-friendly form. This is done by
1.244 +utilizing the *formatting* string provided at log time. The logger
1.245 +simply calls the *format* method of the formatting string, passing the
1.246 +dict containing the message's fields.
1.247 +
1.248 +When *mach* is used in a terminal that supports it, the logging facility
1.249 +also supports terminal features such as colorization. This is done
1.250 +automatically in the logging layer - there is no need to control this at
1.251 +logging time.
1.252 +
1.253 +In addition, messages intended for humans typically prepends every line
1.254 +with the time passed since the application started.
1.255 +
1.256 +Logging HOWTO
1.257 +-------------
1.258 +
1.259 +Structured logging piggybacks on top of Python's built-in logging
1.260 +infrastructure provided by the *logging* package. We accomplish this by
1.261 +taking advantage of *logging.Logger.log()*'s *extra* argument. To this
1.262 +argument, we pass a dict with the fields *action* and *params*. These
1.263 +are the string *action* and dict of message fields, respectively. The
1.264 +formatting string is passed as the *msg* argument, like normal.
1.265 +
1.266 +If you were logging to a logger directly, you would do something like:
1.267 +
1.268 + logger.log(logging.INFO, 'My name is {name}',
1.269 + extra={'action': 'my_name', 'params': {'name': 'Gregory'}})
1.270 +
1.271 +The JSON logging would produce something like:
1.272 +
1.273 + [1339985554.306338, "my_name", {"name": "Gregory"}]
1.274 +
1.275 +Human logging would produce something like:
1.276 +
1.277 + 0.52 My name is Gregory
1.278 +
1.279 +Since there is a lot of complexity using logger.log directly, it is
1.280 +recommended to go through a wrapping layer that hides part of the
1.281 +complexity for you. The easiest way to do this is by utilizing the
1.282 +LoggingMixin:
1.283 +
1.284 + import logging
1.285 + from mach.mixin.logging import LoggingMixin
1.286 +
1.287 + class MyClass(LoggingMixin):
1.288 + def foo(self):
1.289 + self.log(logging.INFO, 'foo_start', {'bar': True},
1.290 + 'Foo performed. Bar: {bar}')
1.291 +
1.292 +Entry Points
1.293 +============
1.294 +
1.295 +It is possible to use setuptools' entry points to load commands
1.296 +directly from python packages. A mach entry point is a function which
1.297 +returns a list of files or directories containing mach command
1.298 +providers. e.g.::
1.299 +
1.300 + def list_providers():
1.301 + providers = []
1.302 + here = os.path.abspath(os.path.dirname(__file__))
1.303 + for p in os.listdir(here):
1.304 + if p.endswith('.py'):
1.305 + providers.append(os.path.join(here, p))
1.306 + return providers
1.307 +
1.308 +See http://pythonhosted.org/setuptools/setuptools.html#dynamic-discovery-of-services-and-plugins
1.309 +for more information on creating an entry point. To search for entry
1.310 +point plugins, you can call *load_commands_from_entry_point*. This
1.311 +takes a single parameter called *group*. This is the name of the entry
1.312 +point group to load and defaults to ``mach.providers``. e.g.::
1.313 +
1.314 + mach.load_commands_from_entry_point("mach.external.providers")
1.315 +
1.316 +Adding Global Arguments
1.317 +=======================
1.318 +
1.319 +Arguments to mach commands are usually command-specific. However,
1.320 +mach ships with a handful of global arguments that apply to all
1.321 +commands.
1.322 +
1.323 +It is possible to extend the list of global arguments. In your
1.324 +*mach driver*, simply call ``add_global_argument()`` on your
1.325 +``mach.main.Mach`` instance. e.g.::
1.326 +
1.327 + mach = mach.main.Mach(os.getcwd())
1.328 +
1.329 + # Will allow --example to be specified on every mach command.
1.330 + mach.add_global_argument('--example', action='store_true',
1.331 + help='Demonstrate an example global argument.')
