Please is designed to support interleaving custom build commands as first-class citizens in the system. This describes some of the concepts to bear in mind when writing rules of your own.
The documentation for genrule() and gentest() may be of use as well, as that explains the available arguments and what they do.
Here is an example of a build rule. It takes in example/file.txt
from the
source tree and counts the number of words within:
# //example:word_count
genrule(
name = "word_count",
srcs = ["file.txt"],
outs = ["file.wordcount"],
cmd = "wc $SRCS > $OUT",
)
Please sets up $SRCS
and $OUT
based on the parameters we passed
to genrule()
, then executes our command. When this rule builds, it will output
plz-out/gen/example/file.wordcount
. For more information on how this works, see
the sources and outputs sections below.
A core concept of Please is to try to isolate execution so we have a good concept of
correctness for each action. To this end each rule builds in its own temporary directory
(under plz-out/tmp
) containing only files defined as inputs to the rule.
This means that you can't access any other files in your source tree from within the rule.
If you want to have a look at what it looks like in there, plz build --shell
will prepare a target for building and open up a shell into the directory, where you can
look around and try running commands by hand.
Some aspects of this environment are different to normal - for example rules are given a small, deterministic set of environment variables. They shouldn't need to read anything outside this directory or not described by those variables. See the Passing environment variables section of this page for more information on passing variables from the host machine to the rule.
Sources are the inputs to your rule - typically source code, but like any other inputs, they can also be build
labels. This is defined in the srcs
argument.
All sources are linked into the build directory - for performance reasons they aren't copied so you shouldn't modify them in-place.
Sources can be both files and directories, however it can be useful to use globs for sources e.g. to include all files with a given extension. See the built ins for more information on this.
Sources can be referred through the $SRCS
environment variable. If there's only
one, $SRC
will be defined too.
Sources can also be named, e.g.
srcs = {
'srcs': ['my_source.py'],
'resources': ['something.txt'],
},
In that case they can be accessed separately through $SRCS_SRCS
and
$SRCS_RESOURCES
.
Rules must define their outputs explicitly. Only these files will be saved to plz-out and made available to other rules.
The output location within plz-out depends on whether the rule is marked as binary. Binary
rules output to plz-out/bin/...
and non-binary rules output to
plz-out/gen/...
.
Like sources, outputs can be both files and directories. The simplest way to define outputs on a rule is as a list:
outs = [
"path/to/a/directory",
"foo.a",
],
The $OUT
and $OUTS
variables will be set much like sources.
Much like sources, outs
can also be a dictionary. In this case, the outputs are
considered named outputs:
genrule(
name = "my_rule",
...
outs = {
'src': ['src'],
'lib': ['lib'],
},
)
The variables $OUTS_SRC
and $OUTS_LIB
will be set. These outputs can then
be depended on individually from other rules:
genrule(
...
srcs = [":my_rule|srcs"],
deps = ["my_rule|libs"],
...
)
Sometimes it can be hard to reason about the exact outputs of a rule before actually building it. Output directories allow you to specify a folder; anything that ends up in that folder will become an output of the rule. The folder's path will not constitute part of the output path of the file.
Imagine we're generating code. The code generation tool parses some sources and based on the symbols it finds, produces a set of outputs e.g. mock generation:
genrule(
name = "mocks",
srcs = ["interfaces.go"],
tools = {
"mockgen": ["mockgen"],
},
cmd = "$TOOLS_MOCKGEN ... $SRCS -o $OUT"
outs = ???,
)
It's difficult to know ahead of time which files mockgen is going to generate. It's dependent on
the source code within interfaces.go
. We can use output directories for this:
genrule(
name = "mocks",
srcs = ["interfaces.go"],
tools = {
"mockgen": ["mockgen"],
},
cmd = "mkdir _out && $TOOLS_MOCKGEN ... $SRCS -o _out"
output_dirs = ["_out"],
)
If mockgen generates foo_mocks.go
and bar/bar_mocks.go
, these will
become outputs of the rules as if they were defines as part of the outs
list:
Build finished; total time 230ms, incrementality 100.0%. Outputs:
//:mocks:
plz-out/gen/foo_mocks.go
plz-out/gen/bar
If instead of the folder plz-out/gen/bar
, it's preferable to have the output as
plz-out/gen/bar/bar_mocks.go
, you may add **
to the output directory:
genrule(
...
output_dirs = ["_out/**"],
)
This can be useful when these files are going to end up as $SRCS
in another rule.
Some tools don't deal well with directories e.g. javac
in java_library()
Dependencies are other things you need in order to build - e.g. other code that your rule depends on.
These are also linked into the build directory, so the difference between sources and dependencies can seem arbitrary, but for some internal functions it's an important distinction to know how rules see the things they'll consume.
For example, the go compiler expects you to explicitly pass your sources (e.g.
go tool compile foo.go bar.go
), however it discovers its dependencies through
configuration (e.g. by looking at the GOPATH).
As such, dependencies don't have any corresponding environment variables associated with them.
Tools are the things that a rule uses to build with. They can refer either to other rules within the repo, or system-level binaries, for example:
tools = [
'curl',
'//path/to:tool',
],
In this example, curl
is resolved on the system using the PATH
variable defined in your .plzconfig file. //path/to:tool
will be built first and used from its output location.
NB: For gentest()
, there's also test_tools
which behaves exactly the same,
except is made available to test_cmd
instead.
Tools are not copied into the build directory, you can access them using the $TOOL
or $TOOLS
environment variables. They can also be named in a similar manner to
sources.
Note that since tools aren't copied, they can be a source of nondeterminism if you make use of other outputs that happen to be located near them. In order to ensure correctness you should make sure that you only run the tool itself and don't access neighbouring outputs.
The distinction between tools and other sources or dependencies is very important when using Please to cross-compile to a different target architecture. Tools will always be used from the host architecture (since they must be executed locally) whereas other dependencies will be for the target.
The command that you run is of course the core part of the rule. It can be passed to
genrule
in three formats:
' && '.join(cmd)
opt
vs.
dbg
but arbitrary names can be given and specified with plz build -c name
.There are various special sequence replacements that the rule is subject to:
$(location //path/to:target)
expands to the location of the given build rule,
which must have a single output only.$(locations //path/to:target)
expands to the locations of the outputs of the
given build rule, which can have any number of outputs.$(dir //path/to:target)
expands to the directory containing the outputs of the
given label$(out_dir //path/to:target)
expands to the directory containing the outputs of the
given label with the preceding plz-out/{gen|bin}$(exe //path/to:target)
expands to a command to run the output of the given
target. The rule must be marked as binary.$(out_exe //path/to:target)
expands to a command to run the output of the given
target with the precdeing plz-out/{gen|bin}. The rule must be marked as binary.$(out_location //path_to:target)
expands to the output of the given build rule,
with the preceding plz-out/{gen|bin}$(out_locations //path_to:target)
expands to the locations of the outputs of the
given build rule, with the preceding plz-out/{gen|bin}. The rule can have any number of outputs.$(hash //path/to:target)
expands to a hash of the outputs of that target.$(worker //path/to:target)
invokes the given target as a
persistent worker. See the linked page for more details on
how to use them.Please executes the build command in isolation. Typically, this means that rules cannot access files and environment variables that have not explcitly been made available to that rule.
The following environment variables are set by please before executing your command:
ARCH
: architecture of the system, eg. amd64OS
: current operating system (linux, darwin, etc).PATH
: usual PATH environment variable as defined in your .plzconfigTMP_DIR
: the temporary directory you're compiling within.HOME
: also set to the temporary directory you're compiling within.NAME
: the name of the rule.
SRCS
: the sources of your ruleOUTS
: the outputs of your rulePKG
: the path to the package containing this rulePKG_DIR
: Similar to PKG
but always contains a path (specifically .
if the rule is in the root of the repo).NAME
: the name of this build ruleOUT
: the output of this rule. Only present when there is only one output.SRC
: the source of this rule. Only present when there is only one source.SRCS_<suffix>
: Present when you've defined named sources on a rule. Each group
creates one of these these variables with paths to those sources.TOOLS
: Any tools defined on the rule.TOOL
: Available on any rule that defines a single tool only.SECRETS
: If any secrets are defined on the rule, these are the paths to them.By default, no environment variables from the host machine are available to rules. To make a
variable available to the rule, set the pass_env
parameter on genrule()
and gentest()
.
These variables will be made available to both cmd
and test_cmd
. They
will also contribute to the rule hash, so if they change, the rule will be re-built and tested as
appropriate.
NB: It's also possible to set environment variables to be passed globally to all rules in
.plzconfig
. See the build section in the config
documentation for more information.
Rules can define a list of secrets that they want access to. These are all absolute paths
(beginning with /
or ~
and aren't copied to the build directory;
instead they can be located using the environment variable $SECRETS
.
They're useful for things like signing or access keys that you don't want to check into
version control but still might be necessary for building some rules.
These don't contribute to the key used to retrieve outputs from the cache; this means it's possible for one machine to build a target with the secret and then share the output with others.
Some tools require resources to run. For example SDKs commonly have their compiler (and/or runtime) in a directory next to the SDK libraries. It can be hard to write a rule to execute this binary in this context. This is because binary rules have a single output whereas here we require the whole folder.
For example the java JDK has the following structure:
.../jdk
|-- bin
| |-- java
| |-- javac
| |-- ...
|-- lib
| |-- ...
...
Using entry points, javac and java can be defined as such:
genrule(
name = "jdk",
cmd = "unzip jdk.zip -o $OUT"
outs = ["out"],
entry_points = {
"java": "jdk/bin/java",
"javac": "jdk/bin/javac",
},
)
These entry points can then be used as tools by other rules, follwing a similar syntax to named outputs:
genrule(
name = "java_lib",
cmd = "$TOOLS_JAVAC ...",
tools = {
"javac": ":jdk|javac",
},
)
As well as genrule()
, there's also gentest() which defines tests.
Test rules are very similar to other build rules in that they have a build step. This build step usually produces a
test binary which is then executed in the test step as defined by test_cmd
:
gentest(
name = "some_test",
...
cmd = "$TOOLS $SRCS -o $OUT", # compiles a binary that we can later run
tools = [CONFIG.SOME_COMPILER], # Some sort of compiler e.g. gcc
outs = ["test"],
test_cmd = "$TEST > $RESULTS_FILE", # Execute the test. $TEST is set to the output of cmd
)
In this example, this rule will generate test
, a binary containing our compiled tests. We then execute
this in test_cmd. The test output is piped to test.results
as defined by the $RESULTS_FILE
variable.
For more information on testing in general, see Testing with Please.