michael@0: ==== michael@0: mach michael@0: ==== michael@0: michael@0: Mach (German for *do*) is a generic command dispatcher for the command michael@0: line. michael@0: michael@0: To use mach, you install the mach core (a Python package), create an michael@0: executable *driver* script (named whatever you want), and write mach michael@0: commands. When the *driver* is executed, mach dispatches to the michael@0: requested command handler automatically. michael@0: michael@0: Features michael@0: ======== michael@0: michael@0: On a high level, mach is similar to using argparse with subparsers (for michael@0: command handling). When you dig deeper, mach offers a number of michael@0: additional features: michael@0: michael@0: Distributed command definitions michael@0: With optparse/argparse, you have to define your commands on a central michael@0: parser instance. With mach, you annotate your command methods with michael@0: decorators and mach finds and dispatches to them automatically. michael@0: michael@0: Command categories michael@0: Mach commands can be grouped into categories when displayed in help. michael@0: This is currently not possible with argparse. michael@0: michael@0: Logging management michael@0: Mach provides a facility for logging (both classical text and michael@0: structured) that is available to any command handler. michael@0: michael@0: Settings files michael@0: Mach provides a facility for reading settings from an ini-like file michael@0: format. michael@0: michael@0: Components michael@0: ========== michael@0: michael@0: Mach is conceptually composed of the following components: michael@0: michael@0: core michael@0: The mach core is the core code powering mach. This is a Python package michael@0: that contains all the business logic that makes mach work. The mach michael@0: core is common to all mach deployments. michael@0: michael@0: commands michael@0: These are what mach dispatches to. Commands are simply Python methods michael@0: registered as command names. The set of commands is unique to the michael@0: environment mach is deployed in. michael@0: michael@0: driver michael@0: The *driver* is the entry-point to mach. It is simply an executable michael@0: script that loads the mach core, tells it where commands can be found, michael@0: then asks the mach core to handle the current request. The driver is michael@0: unique to the deployed environment. But, it's usually based on an michael@0: example from this source tree. michael@0: michael@0: Project State michael@0: ============= michael@0: michael@0: mach was originally written as a command dispatching framework to aid michael@0: Firefox development. While the code is mostly generic, there are still michael@0: some pieces that closely tie it to Mozilla/Firefox. The goal is for michael@0: these to eventually be removed and replaced with generic features so michael@0: mach is suitable for anybody to use. Until then, mach may not be the michael@0: best fit for you. michael@0: michael@0: Implementing Commands michael@0: --------------------- michael@0: michael@0: Mach commands are defined via Python decorators. michael@0: michael@0: All the relevant decorators are defined in the *mach.decorators* module. michael@0: The important decorators are as follows: michael@0: michael@0: CommandProvider michael@0: A class decorator that denotes that a class contains mach michael@0: commands. The decorator takes no arguments. michael@0: michael@0: Command michael@0: A method decorator that denotes that the method should be called when michael@0: the specified command is requested. The decorator takes a command name michael@0: as its first argument and a number of additional arguments to michael@0: configure the behavior of the command. michael@0: michael@0: CommandArgument michael@0: A method decorator that defines an argument to the command. Its michael@0: arguments are essentially proxied to ArgumentParser.add_argument() michael@0: michael@0: Classes with the *@CommandProvider* decorator *must* have an *__init__* michael@0: method that accepts 1 or 2 arguments. If it accepts 2 arguments, the michael@0: 2nd argument will be a *MachCommandContext* instance. This is just a named michael@0: tuple containing references to objects provided by the mach driver. michael@0: michael@0: Here is a complete example:: michael@0: michael@0: from mach.decorators import ( michael@0: CommandArgument, michael@0: CommandProvider, michael@0: Command, michael@0: ) michael@0: michael@0: @CommandProvider michael@0: class MyClass(object): michael@0: @Command('doit', help='Do ALL OF THE THINGS.') michael@0: @CommandArgument('--force', '-f', action='store_true', michael@0: help='Force doing it.') michael@0: def doit(self, force=False): michael@0: # Do stuff here. michael@0: michael@0: When the module is loaded, the decorators tell mach about all handlers. michael@0: When mach runs, it takes the assembled metadata from these handlers and michael@0: hooks it up to the command line driver. Under the hood, arguments passed michael@0: to the decorators are being used to help mach parse command arguments, michael@0: formulate arguments to the methods, etc. See the documentation in the michael@0: *mach.base* module for more. michael@0: michael@0: The Python modules defining mach commands do not need to live inside the michael@0: main mach source tree. michael@0: michael@0: Conditionally Filtering Commands michael@0: -------------------------------- michael@0: michael@0: Sometimes it might only make sense to run a command given a certain michael@0: context. For example, running tests only makes sense if the product michael@0: they are testing has been built, and said build is available. To make michael@0: sure a command is only runnable from within a correct context, you can michael@0: define a series of conditions on the *Command* decorator. michael@0: michael@0: A condition is simply a function that takes an instance of the michael@0: *CommandProvider* class as an argument, and returns True or False. If michael@0: any of the conditions defined on a command return False, the command michael@0: will not be runnable. The doc string of a condition function is used in michael@0: error messages, to explain why the command cannot currently be run. michael@0: michael@0: Here is an example: michael@0: michael@0: from mach.decorators import ( michael@0: CommandProvider, michael@0: Command, michael@0: ) michael@0: michael@0: def build_available(cls): michael@0: """The build needs to be available.""" michael@0: return cls.build_path is not None michael@0: michael@0: @CommandProvider michael@0: class MyClass(MachCommandBase): michael@0: def __init__(self, build_path=None): michael@0: self.build_path = build_path michael@0: michael@0: @Command('run_tests', conditions=[build_available]) michael@0: def run_tests(self): michael@0: # Do stuff here. michael@0: michael@0: It is important to make sure that any state needed by the condition is michael@0: available to instances of the command provider. michael@0: michael@0: By default all commands without any conditions applied will be runnable, michael@0: but it is possible to change this behaviour by setting *require_conditions* michael@0: to True: michael@0: michael@0: m = mach.main.Mach() michael@0: m.require_conditions = True michael@0: michael@0: Minimizing Code in Commands michael@0: --------------------------- michael@0: michael@0: Mach command modules, classes, and methods work best when they are michael@0: minimal dispatchers. The reason is import bloat. Currently, the mach michael@0: core needs to import every Python file potentially containing mach michael@0: commands for every command invocation. If you have dozens of commands or michael@0: commands in modules that import a lot of Python code, these imports michael@0: could slow mach down and waste memory. michael@0: michael@0: It is thus recommended that mach modules, classes, and methods do as michael@0: little work as possible. Ideally the module should only import from michael@0: the *mach* package. If you need external modules, you should import them michael@0: from within the command method. michael@0: michael@0: To keep code size small, the body of a command method should be limited michael@0: to: michael@0: michael@0: 1. Obtaining user input (parsing arguments, prompting, etc) michael@0: 2. Calling into some other Python package michael@0: 3. Formatting output michael@0: michael@0: Of course, these recommendations can be ignored if you want to risk michael@0: slower performance. michael@0: michael@0: In the future, the mach driver may cache the dispatching information or michael@0: have it intelligently loaded to facilitate lazy loading. michael@0: michael@0: Logging michael@0: ======= michael@0: michael@0: Mach configures a built-in logging facility so commands can easily log michael@0: data. michael@0: michael@0: What sets the logging facility apart from most loggers you've seen is michael@0: that it encourages structured logging. Instead of conventional logging michael@0: where simple strings are logged, the internal logging mechanism logs all michael@0: events with the following pieces of information: michael@0: michael@0: * A string *action* michael@0: * A dict of log message fields michael@0: * A formatting string michael@0: michael@0: Essentially, instead of assembling a human-readable string at michael@0: logging-time, you create an object holding all the pieces of data that michael@0: will constitute your logged event. For each unique type of logged event, michael@0: you assign an *action* name. michael@0: michael@0: Depending on how logging is configured, your logged event could get michael@0: written a couple of different ways. michael@0: michael@0: JSON Logging michael@0: ------------ michael@0: michael@0: Where machines are the intended target of the logging data, a JSON michael@0: logger is configured. The JSON logger assembles an array consisting of michael@0: the following elements: michael@0: michael@0: * Decimal wall clock time in seconds since UNIX epoch michael@0: * String *action* of message michael@0: * Object with structured message data michael@0: michael@0: The JSON-serialized array is written to a configured file handle. michael@0: Consumers of this logging stream can just perform a readline() then feed michael@0: that into a JSON deserializer to reconstruct the original logged michael@0: message. They can key off the *action* element to determine how to michael@0: process individual events. There is no need to invent a parser. michael@0: Convenient, isn't it? michael@0: michael@0: Logging for Humans michael@0: ------------------ michael@0: michael@0: Where humans are the intended consumer of a log message, the structured michael@0: log message are converted to more human-friendly form. This is done by michael@0: utilizing the *formatting* string provided at log time. The logger michael@0: simply calls the *format* method of the formatting string, passing the michael@0: dict containing the message's fields. michael@0: michael@0: When *mach* is used in a terminal that supports it, the logging facility michael@0: also supports terminal features such as colorization. This is done michael@0: automatically in the logging layer - there is no need to control this at michael@0: logging time. michael@0: michael@0: In addition, messages intended for humans typically prepends every line michael@0: with the time passed since the application started. michael@0: michael@0: Logging HOWTO michael@0: ------------- michael@0: michael@0: Structured logging piggybacks on top of Python's built-in logging michael@0: infrastructure provided by the *logging* package. We accomplish this by michael@0: taking advantage of *logging.Logger.log()*'s *extra* argument. To this michael@0: argument, we pass a dict with the fields *action* and *params*. These michael@0: are the string *action* and dict of message fields, respectively. The michael@0: formatting string is passed as the *msg* argument, like normal. michael@0: michael@0: If you were logging to a logger directly, you would do something like: michael@0: michael@0: logger.log(logging.INFO, 'My name is {name}', michael@0: extra={'action': 'my_name', 'params': {'name': 'Gregory'}}) michael@0: michael@0: The JSON logging would produce something like: michael@0: michael@0: [1339985554.306338, "my_name", {"name": "Gregory"}] michael@0: michael@0: Human logging would produce something like: michael@0: michael@0: 0.52 My name is Gregory michael@0: michael@0: Since there is a lot of complexity using logger.log directly, it is michael@0: recommended to go through a wrapping layer that hides part of the michael@0: complexity for you. The easiest way to do this is by utilizing the michael@0: LoggingMixin: michael@0: michael@0: import logging michael@0: from mach.mixin.logging import LoggingMixin michael@0: michael@0: class MyClass(LoggingMixin): michael@0: def foo(self): michael@0: self.log(logging.INFO, 'foo_start', {'bar': True}, michael@0: 'Foo performed. Bar: {bar}') michael@0: michael@0: Entry Points michael@0: ============ michael@0: michael@0: It is possible to use setuptools' entry points to load commands michael@0: directly from python packages. A mach entry point is a function which michael@0: returns a list of files or directories containing mach command michael@0: providers. e.g.:: michael@0: michael@0: def list_providers(): michael@0: providers = [] michael@0: here = os.path.abspath(os.path.dirname(__file__)) michael@0: for p in os.listdir(here): michael@0: if p.endswith('.py'): michael@0: providers.append(os.path.join(here, p)) michael@0: return providers michael@0: michael@0: See http://pythonhosted.org/setuptools/setuptools.html#dynamic-discovery-of-services-and-plugins michael@0: for more information on creating an entry point. To search for entry michael@0: point plugins, you can call *load_commands_from_entry_point*. This michael@0: takes a single parameter called *group*. This is the name of the entry michael@0: point group to load and defaults to ``mach.providers``. e.g.:: michael@0: michael@0: mach.load_commands_from_entry_point("mach.external.providers") michael@0: michael@0: Adding Global Arguments michael@0: ======================= michael@0: michael@0: Arguments to mach commands are usually command-specific. However, michael@0: mach ships with a handful of global arguments that apply to all michael@0: commands. michael@0: michael@0: It is possible to extend the list of global arguments. In your michael@0: *mach driver*, simply call ``add_global_argument()`` on your michael@0: ``mach.main.Mach`` instance. e.g.:: michael@0: michael@0: mach = mach.main.Mach(os.getcwd()) michael@0: michael@0: # Will allow --example to be specified on every mach command. michael@0: mach.add_global_argument('--example', action='store_true', michael@0: help='Demonstrate an example global argument.')