Spacefile YAML reference

The YAML syntax supported by Space is a subset of the standard. A node without any value is ignored, except for _env variables which then are treated as empty (re)declarations of the variable.
Nodes right below an _env node are subject to special treatment and their values are evaluated by Bash,
while other nodes are not evaluated with variable substitution, etc.

To set a node to an empty string:

   _env:  
       - varname: ""  

however, the following:

   _env:  
       - varname:  

Will implicitly result in:

   _env:  
       - varname: ${varname-}  

which is useful for making sure that the variable is declared,
but also not overwriting it.

To unset an _env variable use !unset:

   _env:  
       - varname: !unset  

This will have Bash unset the varname variable.

One can also provide meta data for the variable by doing:

   _env:  
       - varname:  
            value: Some value  
            title: This is a special variable  
            desc: |  
                It has such meaning, I can't even begin  
                to explain its importance.  

Using the object definition of an environment variable gives you abilities to attach meta data to the variable for describing it. Also this is useful for auto completion, more on that later.
If value: node is left out, the variable will implicitly be assigned it self as default value, as:

   _env:  
       - varname:  
            value: ${varname-}  

Node environment variables which are wrapped in single quotes will not be Bash evaluated at the parse stage. The quotes are automatically removed.

Node environment variables which are wrapped in double quotes will be Bash evaluated at the parse stage, just as values that are unquoted. The quotes are automatically removed.

Using environment variables in YAML will be properly substituted, if not escaped. However subshells $(...) are not allowed and the $ will be filtered out.

NOTE: if $(...) is quoted in single quotes it will not be filtered out and the subcommand will be executed at runtime!

Valid variable names

In Bash and other shells variable names can contain the characters a-zA-Z0-9_, but cannot begin with a number (0-9).

In Space, the above rules apply but also variables declared in the YAML definitions cannot start with an underscore. This is to protect the inner workings of Space while the YAML is being processed, because variables in shell have very “generous” scopes.

If using any other invalid characters for variable names declared in the YAML, Space might silently ignore them (and not throw an error) because the YAML parser is limited.

Strings and quoting

Unquoted and double quoted values will be parameter expanded by Bash during parsing, while single quoted lines will be treated as raw strings and evaluated later, at run time.

The following node’s value will be parameter expanded by Bash during YAML parsing:

_env:
     - key: "$USER"

Meaning that key will immediately be set with the current username given by the environment.
In the following node the second $USER will not be expanded immediately during YAML parsing:

_env:
     - key: "YAML was processed by: $USER, following user at run time is: \$USER."

Note the escaped dollar sign, so it does not evaluate during YAML parsing step.

This other example on the other hand will be read as a raw string, since it is single quoted:

_env
     - key: '$USER'

It will be processed later at run time.

Other examples:

_env:
    - key1: Regular string can do ${USER}  
    - key2: "Escaped Regular string can do ${USER} and \"${USER}\""  
    - key3: 'Single escaped string will not expand ${USER}'  

Multiline values must start with a single > or |. Using a > will collapse newlines into spaces, while using a | will preserve newlines. If the > or | is immediately followed by a -, then there will be no trailing newline after the last row.

If the first line begins with a single quote ', then the last line must also end with a single quote, and then variables will not get expanded by Bash while parsing.

_env:
     - key1: |  
             Multiline  
             string,  
             with new lines kept.  
             Also last line ends with \n  
  
     - key2: |-  
             Multiline  
             string,  
             with new lines kept.  
             Thanks to '-' this line does not end with \n  
  
     - key3: >  
             These lines  
             will be one  
             one long line,  
             where each line will be  
             space separated. Also the single line will have \n  
  
     - key4: >-  
             These lines  
             will be one  
             one long line,  
             where each line will be  
             space separated. But the single line will not have \n thanks to '-'  
  
     - key5: !nospace >  
             Now these lines  
             will be joined  
             but with no spaces in between!  
             But line will end with \n  
  
     - key6: !nospace >-  
             And these lines  
             will be joined  
             also with no spaces in between!  
             But line will not end with \n thanks to the '-'.  
  
     - key7: |  
             ${another_key}Another line  
             Yet another line here.  
   
     - key8: |  
          Right now user is: $USER.  
          But at run time user would be: \$USER  

Multilines can also do variable expansion just as regular lines.
They can also start and end with single (or double but that is same as no) quotes.

_env:
    str: |  
        'Multiline  
        within single  
        quotes does not expand $USER at parse time.'  

To add empty lines within a multiline string we must have spaces as indentation (here shown with dots), this is necessary due to restrictions in the Space YAML parser.

_env:
    str: |  
        Multiline  
  
        With some empty lines  
        Thanks to the spaces  
  
  
        Thanks!  

Lists and associative arrays

Note that nodes and their values that or not environment variables, that is, not right below an _env node are not evaluated, stripped of quotes etc.

A node can have child nodes, and then it becomes an associative array.

   node:  
       keya: aaa  
       keyb: bbb  

Space YAML supports a kind of list, it is an associative array with numeric keys. When fetching all keys for a list the key names are guaranteed to come sorted so the order of the list is preserved if keys are numbers.

   node:  
      0: first item  
      1: second item  

Which is the same as:

   node:  
      - first item  
      - second item  

Space does not support inline lists or objects, such as:

    node: [1,2,3,4]
    node: {"a":1, "b":2}

Such an entry will throw an error and Space will exit.

Special keys

All nodes with names beginning with an underscore are treated as hidden nodes and cannot autocomplete from the command line. They also might have special meaning to Space.
Also when doing regular expressions on node targets, underscored target names are ignored, if the regular expression pattern does not use underscore.

This will ignore node names beginning with underscore:

space "/.*/"

This will include node names beginning with underscore or no underscore:

space "/_?.*/"
  • _info: Used purely for information and documentation about a node.
   nodes:  
       my1:  
           _info:  
               title: This is my node no 1  
               desc: |  
                   Let's have a  
                   multiline description  
                   about it.  
  • _env: List of variables to set below a node. Variables could reference other variables. Referenced variables must have been set on a node above or if on the same node then a sub _env is necessary for that dependent variable.
   nodes:  
       _env:  
           - TEXT: text about something  
       my1:  
           _env:  
               - A: some $TEXT  
               - B: A is: $A  
  • _source: Source a shell script file.
    node:
        _source:
            - a.sh
            - b.sh

Note that when sourcing shell files only the functions are of interested to Space, anything outside of function bodies is ignored.

Preprocessor

Preprocessing variable assignments, usage and directives:

  • @{DIR}:
    Variable containing the directory of the YAML file currently being processed.

  • @{DIRNAME}:
    Variable containing the basename of the directory of the YAML file currently being processed.

  • @{PARENT}
    Variable containing the current parent node name. For list items, it is the list objects parent node name.

  • @{PARENTPATH}
    Variable containing the full current parent node path. For list items, it is the list objects parent node path.

  • @{varname}
    User defined variable.

  • @varname:
    If any value is defined, it will set varname.
    If followed by a trailing space variable will be set to empty.
    If no value is defined nor is it followed by a trailing space it will unset varname

  • @varname:-
    Any value will set varname only if it the variable is unset or empty

  • @varname:+
    Append to varname if it already contains data

  • @{varname-default value if unset} or @{varname:-default value if empty}: Use @{varname} anywhere to have it substituted with the value of the variable. Default value could be another variable from the preprocessor such as @{var2}. Ex: @{var1-@{var2}}

Variables are scoped in the sense that for each include a new variable instance is created for the same variable name. This means that overriding a variable in an include will not override the variable in the outer scope.

However, is you use upper case letters [A-Z0-9_] as the variable name the variable will be considered global from the scope where it was first declared, and nested scopes could then override the variable value so that the values can be allowed to “bubble” upwards.

Preprocessor keywords

The following keywords are functions and cannot be used as preprocessor variables:

  • @assert: nonempty @{variable}: Do nonempty assertion of space preprocessor variable expression.

  • @prompt: var1 Text to prompt user with: This will output the text and read input from stdin which will be stored in preprocess variable “var1”.
    Use @prompt:- to only prompt if var1 is unset or empty.
    One exception happens when using prompt in the preprocessor: whenever preprocess output is cached, so will the prompted value be. The next time it runs the user will not be prompted because Space will be running a cached version.
    Please refer to the caching options for controlling that behavior from the command line: -C option.
    A note of warning is that tab auto completion and @prompt should generally not be used together since the prompt will get in the way of the completion process because it outputs to stderr and reads from stdin.

  • @cache: {0,1,2}: Force the caching behaviour of this YAML file. Could be useful to turn off caching when using @prompt, since prompted values are cached.
    A note of warning though is that you should not turn off caching when using tab auto completion because the performance and responsiveness will suffer greatly.

  • @clone: space-sh/ssh space-sh/docker: The clone directive exists for using another module.
    The @clone: keyword can be defined anywhere in the YAML structure. When using the include filters this feature can be leveraged to only clone modules as needed. It is possible to specify multiple arguments in a single @clone: statement.
    Internally @clone translates to _source in preprocessed YAML and points to shell files to be sourced when the YAML node is parsed.
    When cloning, one could specify a module version to use. For example:
    @clone: mymodule:1.2.3 or @clone: mymodule:master
    If no version is specified, Space will try to figure out what version to use according to Space’s own version.

  • @source: @{DIR}/file.sh @{DIR}/file2.sh Internally, this translates to _source: ... in preprocessed YAML and will trigger the source action when the parsed node is loaded.

  • @debug: varname is: @{varname}: Output using “_debug” during preprocess stage, you need to use the -v4 flag to see the debug output.

  • @include: path/ssh.yaml|/some/node/(arg1 arg2): Include the ssh.yaml file and filter on /some/node, also assign the preprocess variables @{1} and @{2} the values arg1 resp arg2. The arguments are space separated and optional. This is a very powerful construct and can turn configuration more dynamic. The filter is also optional.
    To include from within the same file leave out the file name, as: @include |/filter/.
    Absolute paths are also allowed.
    Anything that is indented below the @include is treated as a sub-document that will get injected below every direct child node from the @include. Like a for loop.
    YAML files are normally cached after parsing. It is not recommended to @include files from modules back into user space, since caching can get confused. User space projects can include user space files and modules. Modules should only include it self and other modules.

  • @dotdot: variable: Perform the equivalent of cd .. on the variable containing a node path.

For each server, include all services:

@include: servers.yaml
    server: This is server @{PARENT}
    @include: services.yaml

A filter is ended with slash if the filter is for everything below that node, if the filter does not end with slash it means: get the content of the node and below. The most useful aspect of this is to dynamically build multiline strings. Examples:

Get the string value:

a: a string
A: >-
    @include: |/a

Get a list of strings using .*:

a:
    - a string
    - another string
A: >-
    @include: |/a/.*
  • @include:- file.yaml:
    Adding the dash will ignore a missing YAML file. Also its sub-document will be ignored.

  • @include: [username/]repo:
    Instead of including file names you could name a module, then Space will search for the right module and YAML file to include.

  • setting default values for unset/empty or unset:

    @C: Is: @{var:-unset or empty}
    @D: Is: @{var-unset}
    
  • setting default value options if set:

    @E: Is: @{t-@b}
    
  • setting default value options if not unset or empty:

    @E: Is: @{t:-@b}
    
  • setting variables via the command line:
    It is also possible to set any preprocessor variables using -pvarname= via the command line. These variables are saved with the cache and if they change or are not provided, the cache will be invalidated.

Internal environment variables

There are two special environment variables, both which are available in the YAML build stage and can be referenced.

  • CWD: The full directory path from where the user invoked Space.

  • CWDNAME: The base name of the directory from where the user invoked Space.

Both these variables are write protected and cannot be assigned from within the YAML.

Examples

The following shows some Space variables in practice.

Setup a test Module

For Spacefile.yaml:

1:
    _env:
        - RUN: echo Hello World!

2:
    _env:
        - RUN_ALIAS: /1/

3:
    _env:
        - SPACE_MUTE_EXIT: 1
        - RUN: return 1

4:
    _env:
        - SPACE_ASSERT_EXIT: 1
        - RUN: return 1

5:
    _env:
        - SPACE_REDIR: "<somefile.txt"
        - RUN: wc -l

6:
    _env:
        - SPACE_ENV: "USER HOME=\${HOME}"
        - RUN: echo \$USER \$HOME

7:
    _env:
        - SPACE_ARGS: Hello -- Galaxy!
        - RUN_ALIAS: /1/

8:
    _env:

        - RUN: FUN1 -- going places

9:
    _env:

        - RUN: FUN2

10:
    _env:

        - SPACE_WRAP: FUN_WR
        - RUN: FUN1

11:
    _env:

        - SPACE_OUTER: FUN_OUT
        - SPACE_OUTERARGS: 3
        - RUN: FUN1 -- Hej!

and Spacefile.sh:

FUN1()
{
    echo "$@"
}

FUN2()
{
    SPACE_DEP="FUN3"

    FUN3 "Greetings from FUN2."
}

FUN3()
{
    echo "FUN3 saying: $*"
}

FUN_WR()
{
    SPACE_FN="FUN_WR_IMPL"
}

FUN_WR_IMPL()
{
    echo "I am a wrapped function. This is the number of bytes in the wrapped command:"
    echo "${RUN}" | wc -c
    echo "And this is the wrapped command I would run:"
    echo "${RUN}"
}

FUN_OUT()
{
    local n="${1}"
    echo "This outer function will run the same command ${n} times:"
    for i in $(seq 1 ${n}); do
        _RUN_
    done
}
Experimenting with the test nodes

Now it is possible to experiment with the test Module nodes. Experiment with the -d switch and observe how the exported code changes on each node, in particular in cases where external parameters affect code output.
Some tests require extra steps. Check the following subsections for extra notes on tests that require extra steps.

SPACE_ENV

In the following example, “builduser” is the user name of the user who built the script and “mutable” signals that the HOME variable was evaluated in runtime.

space -f Spacefile.yaml /6/ -d | USER=immutable HOME=mutable sh
builduser mutable

When assigning values in build time, one should always quote the value, if it might contain any spaces, to prevent bad word expansion. The next example illustrates that var3 might contain spaces, so it is quoted.

SPACE_ENV="var1=nospaces var2=\"has spaces\" var3=\"${var3}\" var4=\"\${var4}\""

An unescaped variable name will be evaluated on the spot in build time. However, it is possible to defer the evaluation until run time by escaping it as shown in $var4 above.

In case automatic assignment is wanted, simply list the variable names:

SPACE_ENV="var1 var2"

These variables will be evaluated and assigned values after all modules have been parsed, so a variable that is exported early in this way will still get the value the variable has at the last stage.
To force the evaluation directly assign the variable to itself as such:

SPACE_ENV="var1=\"${var1}\""

To avoid errors about undeclared variables and providing a default value, do:

SPACE_ENV="var1=\"${var1-${var2}}\""
SPACE_ARGS

Argument passing works as regular functions:

space /7/
Hello Galaxy!
space /7/ -- Universe!
Hello Universe!
SPACE_FN

In this example, RUN points to an actual module function.

space /8/ -d
...

space /8/
going places

space /8/ -- Going Places!
Going Places!

Bash auto completion

Space supports the Bash auto completion implementation.
To allow environment variables to be auto completed use the following definition:

_env: 
    - var1:
        value: ${var1-default value}
        values:
            - Value1
            - Value2
            - Value3

This will make it so that space -e[tab] will expand to space -e var1= and space -e var1=[tab][tab] will list the three values provided.

If no value: is defined then the implicit default value: ${var1-} is used, to have the variable be defined and assigned to it self.

The values below the values: node are never automatically used as default values, they are only used for tab completion.

Also auto completion on positional arguments is supported:

_env: 
    - SPACE_ARGS:
        value: -- ${SPACE_ARGS-default value}
        arguments:
            -
                values:
                    - A
                    - B
                    - Aaa
                    - AAa
                    - Ba
            -
                values:
                    - Arg2ThisIs
                    - Arg2ItIs

If no value: is defined then the implicit default value: ${SPACE_ARGS-} is used, to have the variable be defined and assigned to it self.

The values below the values: node are never automatically used as default values, they are only used for tab completion.

Bash dynamic auto completion

Space has a very advanced auto completion system.
Properly configured it can fetch auto complete alternatives from a Docker Engine running on a machine behind a firewall that you need three SSH hops to reach. Really!

_env: 
    - var1:
        value: ${var1-default value}
        completion: /list_all/
    - var2:
        completion: G

_env: - SPACE_ARGS: value: – ${SPACE_ARGS-default value} arguments: - completion: /list_all/ - completion: /list_some/


Add the `completion` node which is the named of a node in the same namespace that
is to fetch the list of options. See the examples section for some nice examples.  
If `completion` is set to `G` then _Space_ will complete using file globbing.

#### Limitations of the Space YAML parser
_Space's_ YAML parser is written in Bash and it supports a subset of the complete
YAML definition.  

There are some known limitations to the parser.  

This will not work:  

```yaml
a:
    - b:
        - s1: hello
        - s2: world
      c:
        - s1: greetings
        - s2: from jupiter

This however will work:

a:
    -
      b:
        - s1: hello
        - s2: world
      c:
        - s1: greetings
        - s2: from jupiter

This will not work:

a:
- item1
- item2

This however will work:

a:
 - item1
 - item2

The Space YAML parsers needs a indentation for list items.

Previous: Space variables

Next: Modules versioning

Edit this page