Spacefile YAML reference
The YAML syntax supported by Space is a subset of the standard. A node without any value is ignored, except for _env
variables which then are treated as empty (re)declarations of the variable.
Nodes right below an _env
node are subject to special treatment and their values are evaluated by Bash,
while other nodes are not evaluated with variable substitution, etc.
To set a node to an empty string:
_env:
- varname: ""
however, the following:
_env:
- varname:
Will implicitly result in:
_env:
- varname: ${varname-}
which is useful for making sure that the variable is declared,
but also not overwriting it.
To unset an _env
variable use !unset
:
_env:
- varname: !unset
This will have Bash unset the varname
variable.
One can also provide meta data for the variable by doing:
_env:
- varname:
value: Some value
title: This is a special variable
desc: |
It has such meaning, I can't even begin
to explain its importance.
Using the object definition of an environment variable gives you
abilities to attach meta data to the variable for describing it.
Also this is useful for auto completion, more on that later.
If value:
node is left out, the variable will implicitly be
assigned it self as default value, as:
_env:
- varname:
value: ${varname-}
Node environment variables which are wrapped in single quotes will not be Bash evaluated at the parse stage. The quotes are automatically removed.
Node environment variables which are wrapped in double quotes will be Bash evaluated at the parse stage, just as values that are unquoted. The quotes are automatically removed.
Using environment variables in YAML will be properly substituted, if not escaped. However subshells $(...)
are not allowed and the $
will be filtered out.
NOTE: if $(...)
is quoted in single quotes it will not be filtered out and the subcommand will be executed at runtime!
Valid variable names
In Bash and other shells variable names can contain the characters a-zA-Z0-9_
, but cannot begin with a number (0-9)
.
In Space, the above rules apply but also variables declared in the YAML definitions cannot start with an underscore. This is to protect the inner workings of Space while the YAML is being processed, because variables in shell have very “generous” scopes.
If using any other invalid characters for variable names declared in the YAML, Space might silently ignore them (and not throw an error) because the YAML parser is limited.
Strings and quoting
Unquoted and double quoted values will be parameter expanded by Bash during parsing, while single quoted lines will be treated as raw strings and evaluated later, at run time.
The following node’s value will be parameter expanded by Bash during YAML parsing:
_env:
- key: "$USER"
Meaning that key
will immediately be set with the current username given by the environment.
In the following node the second $USER
will not be expanded immediately during YAML parsing:
_env:
- key: "YAML was processed by: $USER, following user at run time is: \$USER."
Note the escaped dollar sign, so it does not evaluate during YAML parsing step.
This other example on the other hand will be read as a raw string, since it is single quoted:
_env
- key: '$USER'
It will be processed later at run time.
Other examples:
_env:
- key1: Regular string can do ${USER}
- key2: "Escaped Regular string can do ${USER} and \"${USER}\""
- key3: 'Single escaped string will not expand ${USER}'
Multiline values must start with a single >
or |
. Using a >
will collapse newlines into spaces, while using a |
will preserve newlines. If the >
or |
is immediately followed by a -
, then there will be no trailing newline after the last row.
If the first line begins with a single quote '
, then the last line must also end with a single quote, and then variables will not get expanded by Bash while parsing.
_env:
- key1: |
Multiline
string,
with new lines kept.
Also last line ends with \n
- key2: |-
Multiline
string,
with new lines kept.
Thanks to '-' this line does not end with \n
- key3: >
These lines
will be one
one long line,
where each line will be
space separated. Also the single line will have \n
- key4: >-
These lines
will be one
one long line,
where each line will be
space separated. But the single line will not have \n thanks to '-'
- key5: !nospace >
Now these lines
will be joined
but with no spaces in between!
But line will end with \n
- key6: !nospace >-
And these lines
will be joined
also with no spaces in between!
But line will not end with \n thanks to the '-'.
- key7: |
${another_key}Another line
Yet another line here.
- key8: |
Right now user is: $USER.
But at run time user would be: \$USER
Multilines can also do variable expansion just as regular lines.
They can also start and end with single (or double but that is same as no) quotes.
_env:
str: |
'Multiline
within single
quotes does not expand $USER at parse time.'
To add empty lines within a multiline string we must have spaces as indentation (here shown with dots), this is necessary due to restrictions in the Space YAML parser.
_env:
str: |
Multiline
With some empty lines
Thanks to the spaces
Thanks!
Lists and associative arrays
Note that nodes and their values that or not environment variables
, that is, not right below an _env
node
are not evaluated, stripped of quotes etc.
A node can have child nodes, and then it becomes an associative array.
node:
keya: aaa
keyb: bbb
Space YAML supports a kind of list, it is an associative array with numeric keys. When fetching all keys for a list the key names are guaranteed to come sorted so the order of the list is preserved if keys are numbers.
node:
0: first item
1: second item
Which is the same as:
node:
- first item
- second item
Space does not support inline lists or objects, such as:
node: [1,2,3,4]
node: {"a":1, "b":2}
Such an entry will throw an error and Space will exit.
Special keys
All nodes with names beginning with an underscore are treated as hidden nodes and cannot autocomplete from the command line. They also might have special meaning to Space.
Also when doing regular expressions on node targets, underscored target names are ignored, if the regular expression pattern does not use underscore.
This will ignore node names beginning with underscore:
space "/.*/"
This will include node names beginning with underscore or no underscore:
space "/_?.*/"
_info
: Used purely for information and documentation about a node.
nodes:
my1:
_info:
title: This is my node no 1
desc: |
Let's have a
multiline description
about it.
_env
: List of variables to set below a node. Variables could reference other variables. Referenced variables must have been set on a node above or if on the same node then a sub_env
is necessary for that dependent variable.
nodes:
_env:
- TEXT: text about something
my1:
_env:
- A: some $TEXT
- B: A is: $A
_source
: Source a shell script file.
node:
_source:
- a.sh
- b.sh
Note that when sourcing shell files only the functions are of interested to Space, anything outside of function bodies is ignored.
Preprocessor
Preprocessing variable assignments, usage and directives:
@{DIR}
:
Variable containing the directory of the YAML file currently being processed.@{DIRNAME}
:
Variable containing the basename of the directory of the YAML file currently being processed.@{PARENT}
Variable containing the current parent node name. For list items, it is the list objects parent node name.@{PARENTPATH}
Variable containing the full current parent node path. For list items, it is the list objects parent node path.@{varname}
User defined variable.@varname
:
If any value is defined, it will setvarname
.
If followed by a trailing space variable will be set to empty.
If no value is defined nor is it followed by a trailing space it will unsetvarname
@varname:-
Any value will setvarname
only if it the variable is unset or empty@varname:+
Append tovarname
if it already contains data@{varname-default value if unset}
or@{varname:-default value if empty}
: Use@{varname}
anywhere to have it substituted with the value of the variable. Default value could be another variable from the preprocessor such as@{var2}
. Ex:@{var1-@{var2}}
Variables are scoped in the sense that for each include
a new variable instance
is created for the same variable name. This means that overriding a variable in an include
will not override the variable in the outer scope.
However, is you use upper case letters [A-Z0-9_]
as the variable name the
variable will be considered global from the scope where it was first declared, and
nested scopes could then override the variable value so that the values can be allowed
to “bubble” upwards.
Preprocessor keywords
The following keywords are functions and cannot be used as preprocessor variables:
@assert: nonempty @{variable}
: Do nonempty assertion of space preprocessor variable expression.@prompt: var1 Text to prompt user with
: This will output the text and read input fromstdin
which will be stored in preprocess variable “var1”.
Use@prompt:-
to only prompt if var1 is unset or empty.
One exception happens when usingprompt
in the preprocessor: whenever preprocess output is cached, so will the prompted value be. The next time it runs the user will not be prompted because Space will be running a cached version.
Please refer to the caching options for controlling that behavior from the command line:-C
option.
A note of warning is that tab auto completion and@prompt
should generally not be used together since the prompt will get in the way of the completion process because it outputs tostderr
and reads fromstdin
.@cache: {0,1,2}
: Force the caching behaviour of this YAML file. Could be useful to turn off caching when using@prompt
, since prompted values are cached.
A note of warning though is that you should not turn off caching when using tab auto completion because the performance and responsiveness will suffer greatly.@clone: space-sh/ssh space-sh/docker
: The clone directive exists for using another module.
The@clone:
keyword can be defined anywhere in the YAML structure. When using theinclude
filters this feature can be leveraged to only clone modules as needed. It is possible to specify multiple arguments in a single@clone:
statement.
Internally@clone
translates to_source
in preprocessed YAML and points to shell files to be sourced when the YAML node is parsed.
When cloning, one could specify a module version to use. For example:
@clone: mymodule:1.2.3
or@clone: mymodule:master
If no version is specified, Space will try to figure out what version to use according to Space’s own version.@source: @{DIR}/file.sh @{DIR}/file2.sh
Internally, this translates to_source: ...
in preprocessed YAML and will trigger the source action when the parsed node is loaded.@debug: varname is: @{varname}
: Output using “_debug” during preprocess stage, you need to use the -v4 flag to see the debug output.@include: path/ssh.yaml|/some/node/(arg1 arg2)
: Include the ssh.yaml file and filter on/some/node
, also assign the preprocess variables@{1}
and@{2}
the values arg1 resp arg2. The arguments are space separated and optional. This is a very powerful construct and can turn configuration more dynamic. The filter is also optional.
To include from within the same file leave out the file name, as:@include |/filter/.
Absolute paths are also allowed.
Anything that is indented below the @include is treated as a sub-document that will get injected below every direct child node from the @include. Like a for loop.
YAML files are normally cached after parsing. It is not recommended to@include
files from modules back into user space, since caching can get confused. User space projects can include user space files and modules. Modules should only include it self and other modules.@dotdot: variable
: Perform the equivalent ofcd ..
on the variable containing a node path.
For each server, include all services:
@include: servers.yaml
server: This is server @{PARENT}
@include: services.yaml
A filter is ended with slash if the filter is for everything below that node, if the filter does not end with slash it means: get the content of the node and below. The most useful aspect of this is to dynamically build multiline strings. Examples:
Get the string value:
a: a string
A: >-
@include: |/a
Get a list of strings using .*
:
a:
- a string
- another string
A: >-
@include: |/a/.*
@include:- file.yaml
:
Adding the dash will ignore a missing YAML file. Also its sub-document will be ignored.@include: [username/]repo
:
Instead of including file names you could name a module, then Space will search for the right module and YAML file to include.setting default values for unset/empty or unset:
@C: Is: @{var:-unset or empty} @D: Is: @{var-unset}
setting default value options if set:
@E: Is: @{t-@b}
setting default value options if not unset or empty:
@E: Is: @{t:-@b}
setting variables via the command line:
It is also possible to set any preprocessor variables using-pvarname=
via the command line. These variables are saved with the cache and if they change or are not provided, the cache will be invalidated.
Internal environment variables
There are two special environment variables, both which are available in the YAML build stage and can be referenced.
CWD
: The full directory path from where the user invoked Space.CWDNAME
: The base name of the directory from where the user invoked Space.
Both these variables are write protected and cannot be assigned from within the YAML.
Examples
The following shows some Space variables in practice.
Setup a test Module
For Spacefile.yaml
:
1:
_env:
- RUN: echo Hello World!
2:
_env:
- RUN_ALIAS: /1/
3:
_env:
- SPACE_MUTE_EXIT: 1
- RUN: return 1
4:
_env:
- SPACE_ASSERT_EXIT: 1
- RUN: return 1
5:
_env:
- SPACE_REDIR: "<somefile.txt"
- RUN: wc -l
6:
_env:
- SPACE_ENV: "USER HOME=\${HOME}"
- RUN: echo \$USER \$HOME
7:
_env:
- SPACE_ARGS: Hello -- Galaxy!
- RUN_ALIAS: /1/
8:
_env:
- RUN: FUN1 -- going places
9:
_env:
- RUN: FUN2
10:
_env:
- SPACE_WRAP: FUN_WR
- RUN: FUN1
11:
_env:
- SPACE_OUTER: FUN_OUT
- SPACE_OUTERARGS: 3
- RUN: FUN1 -- Hej!
and Spacefile.sh
:
FUN1()
{
echo "$@"
}
FUN2()
{
SPACE_DEP="FUN3"
FUN3 "Greetings from FUN2."
}
FUN3()
{
echo "FUN3 saying: $*"
}
FUN_WR()
{
SPACE_FN="FUN_WR_IMPL"
}
FUN_WR_IMPL()
{
echo "I am a wrapped function. This is the number of bytes in the wrapped command:"
echo "${RUN}" | wc -c
echo "And this is the wrapped command I would run:"
echo "${RUN}"
}
FUN_OUT()
{
local n="${1}"
echo "This outer function will run the same command ${n} times:"
for i in $(seq 1 ${n}); do
_RUN_
done
}
Experimenting with the test nodes
Now it is possible to experiment with the test Module nodes. Experiment with the -d
switch and observe how the exported code changes on each node, in particular in cases where external parameters affect code output.
Some tests require extra steps. Check the following subsections for extra notes on tests that require extra steps.
SPACE_ENV
In the following example, “builduser” is the user name of the user who built the script and “mutable” signals that the HOME variable was evaluated in runtime.
space -f Spacefile.yaml /6/ -d | USER=immutable HOME=mutable sh
builduser mutable
When assigning values in build time, one should always quote the value, if it might contain any spaces, to prevent bad word expansion. The next example illustrates that var3
might contain spaces, so it is quoted.
SPACE_ENV="var1=nospaces var2=\"has spaces\" var3=\"${var3}\" var4=\"\${var4}\""
An unescaped variable name will be evaluated on the spot in build time. However, it is possible to defer the evaluation until run time by escaping it as shown in $var4
above.
In case automatic assignment is wanted, simply list the variable names:
SPACE_ENV="var1 var2"
These variables will be evaluated and assigned values after all modules have been parsed, so a variable that is exported
early in this way will still get the value the variable has at the last stage.
To force the evaluation directly assign the variable to itself as such:
SPACE_ENV="var1=\"${var1}\""
To avoid errors about undeclared variables and providing a default value, do:
SPACE_ENV="var1=\"${var1-${var2}}\""
SPACE_ARGS
Argument passing works as regular functions:
space /7/
Hello Galaxy!
space /7/ -- Universe!
Hello Universe!
SPACE_FN
In this example, RUN
points to an actual module function.
space /8/ -d
...
space /8/
going places
space /8/ -- Going Places!
Going Places!
Bash auto completion
Space supports the Bash auto completion implementation.
To allow environment variables to be auto completed use the following definition:
_env:
- var1:
value: ${var1-default value}
values:
- Value1
- Value2
- Value3
This will make it so that space -e[tab]
will expand to space -e var1=
and
space -e var1=[tab][tab]
will list the three values provided.
If no value:
is defined then the implicit default value: ${var1-}
is used, to have the variable be defined and assigned to it self.
The values below the values:
node are never automatically used as default values, they are only used for tab completion.
Also auto completion on positional arguments is supported:
_env:
- SPACE_ARGS:
value: -- ${SPACE_ARGS-default value}
arguments:
-
values:
- A
- B
- Aaa
- AAa
- Ba
-
values:
- Arg2ThisIs
- Arg2ItIs
If no value:
is defined then the implicit default value: ${SPACE_ARGS-}
is used, to have the variable be defined and assigned to it self.
The values below the values:
node are never automatically used as default values, they are only used for tab completion.
Bash dynamic auto completion
Space has a very advanced auto completion system.
Properly configured it can fetch auto complete alternatives from a Docker Engine
running on a machine behind a firewall that you need three SSH hops to reach. Really!
_env:
- var1:
value: ${var1-default value}
completion: /list_all/
- var2:
completion: G
_env: - SPACE_ARGS: value: – ${SPACE_ARGS-default value} arguments: - completion: /list_all/ - completion: /list_some/
Add the `completion` node which is the named of a node in the same namespace that
is to fetch the list of options. See the examples section for some nice examples.
If `completion` is set to `G` then _Space_ will complete using file globbing.
#### Limitations of the Space YAML parser
_Space's_ YAML parser is written in Bash and it supports a subset of the complete
YAML definition.
There are some known limitations to the parser.
This will not work:
```yaml
a:
- b:
- s1: hello
- s2: world
c:
- s1: greetings
- s2: from jupiter
This however will work:
a:
-
b:
- s1: hello
- s2: world
c:
- s1: greetings
- s2: from jupiter
This will not work:
a:
- item1
- item2
This however will work:
a:
- item1
- item2
The Space YAML parsers needs a indentation for list items.
Previous: Space variables
Next: Modules versioning
Edit this page