# Whistle basics

This page describes the basics of writing a mapping in Whistle to transform data
from one schema to another.

For a list of built-in Whistle functions, see the
[Whistle built-in function reference](https://docs.cloud.google.com/manufacturing-data-engine/docs/reference/whistle/whistle-builtins.md.txt),
and for a list of MDE-specific functions, see the
[Whistle MDE function reference](https://docs.cloud.google.com/manufacturing-data-engine/docs/reference/whistle/whistle-mde-functions.md.txt).

## Simple mapping

The following section outlines some array outputs.

### Add array outputs

If you use the following Whistle code sample:

```python
Planet[0]: "Earth";
Planet[1]: "Mars";
Planet[2]: "Jupiter";
Moon[0]: "Luna";
```

The output would be the following:

```python
{
  "Moon": [
    "Luna"
  ],
  "Planet": [
    "Earth",
    "Mars",
    "Jupiter"
  ]
}
```

To add another moon, `Io`, before `Luna`, edit your Whistle code as follows:

```python
Planet[0]: "Earth";
Planet[1]: "Mars";
Planet[2]: "Jupiter";
Moon[0]: "Io";
Moon[1]: "Luna";
```

With the previous input, the output will be the following:

```python
{
  "Moon": [
    "Io",
    "Luna"
  ],
  "Planet": [
    "Earth",
    "Mars",
    "Jupiter"
  ]
}
```

## Define and call functions

This section describes call functions.

### Defining functions

A function in Whistle is a set of mappings that produce a JSON object. The function maps a
set of inputs to a set of fields in the resulting JSON object. The following
examples use a function to add structure to the *planets* from the previous section.

```python
Planet[0]: PlanetName_PlanetInfo("Earth");
Planet[1]: PlanetName_PlanetInfo("Mars");
Planet[2]: PlanetName_PlanetInfo("Jupiter");
Moon[0]: MoonName_MoonInfo("Luna");

def PlanetName_PlanetInfo(planetName) {
  name: planetName;
  type: "Planet"
}

def MoonName_MoonInfo(moonName) {
  name: moonName;
  type: "Moon"
}
```

With the previous input, the output would be the following:

```json
{
  "Moon": [
    {
      "name": "Luna",
      "type": "Moon"
    }
  ],
  "Planet": [
    {
      "name": "Earth",
      "type": "Planet"
    },
    {
      "name": "Mars",
      "type": "Planet"
    },
    {
      "name": "Jupiter",
      "type": "Planet"
    }
  ]
}
```

### Calling functions

The following sample demonstrates calling functions, which is similar to C or
Python.

A function call looks like `FunctionName(a, b, c)`.

You can chain function calls by passing the result of one function to the next one. For
example, `SingleParameterFunctionName(FunctionName(a, b, c))`.

Similarly, you can pass the results of a function into a function with multiple parameters.
For example, `MultipleParamFunctionName(FunctionName(a, b, c), d)`.

The following sample function generalizes the function from the previous section
by making the celestial body's `type` an input.

```python
Planet[0]: BodyName_BodyType_BodyInfo("Earth", "Planet");
Planet[1]: BodyName_BodyType_BodyInfo("Mars", "Planet");
Planet[2]: BodyName_BodyType_BodyInfo("Jupiter", "Planet");
Moon[0]: BodyName_BodyType_BodyInfo("Luna", "Moon");

def BodyName_BodyType_BodyInfo(bodyName, bodyType) {
  name: bodyName
  type: bodyType
}
```

With the previous input, the output would be the following:

```json
{
  "Moon": [
    {
      "name": "Luna",
      "type": "Moon"
    }
  ],
  "Planet": [
    {
      "name": "Earth",
      "type": "Planet"
    },
    {
      "name": "Mars",
      "type": "Planet"
    },
    {
      "name": "Jupiter",
      "type": "Planet"
    }
  ]
}
```

### Functions returning a primitive

Functions in Whistle can return arrays, objects, and primitives.

The following sample function sets the `Primitive` field to the number `20`:

    Primitive: Num_DoubleNum(10);

    def Num_DoubleNum(num) 2 * num

With the previous input, the output is the following:

```json
{
  "Primitive": 20
}
```

## Merge semantics

The following steps show how data objects are merged using Whistle.

Given the following mapping:

        Merged: Colour_Colour_MergedColours("red", "blue")

        def Colour_Colour_MergedColours(col1, col2) {
          field1: "hello";

          Colour_Col1(col1);

          Colour_Col2(col2);
        }

        def Colour_Col1(col) {
          colour.first: col;
          colours[1]: col;
        }

        def Colour_Col2(col) {
          colour.second: col;
          colours[1]: col;
        }

The output would be the following:

```json
    {
      "Merged": {
          "colour": {
            "first": "red",
            "second": "blue"
          },
          "colours": [
            "blue"
          ]
          "field1": "hello"
      },
    }
```

Data objects are merged as follows:

- Arrays are concatenated.
- Hardcoded array indexes are preserved, so in the previous example, `blue` overwrites `red`.
- New fields in objects are added.
- Existing fields in objects are merged recursively.
- Primitives are overwritten.
- `null` and empty values don't overwrite existing values.

## Mapping from input data

Given the following source message:

```json
{
  "Planets": [
    {
      "name": "Earth"
    },
    {
      "name": "Mars"
    },
    {
      "name": "Jupiter"
    }
  ],
  "Moons": [
    {
      "name": "Luna"
    }
  ]
}
```

This data is loaded into an input called `$root`. Data loading into an input to
the mapping engine is always in this `$root` input. You can use `$root` inside
functions, but you should avoid doing so because it can signal non-modular mappings.

```python
Planet[0]: BodyName_BodyType_BodyInfo($root.Planets[0], "Planet");
Planet[1]: BodyName_BodyType_BodyInfo($root.Planets[1], "Planet");
Planet[2]: BodyName_BodyType_BodyInfo($root.Planets[2], "Planet");
Moon[0]: BodyName_BodyType_BodyInfo($root.Moons[0], "Moon");

def BodyName_BodyType_BodyInfo(body, bodyType) {
  name: body.name
  type: bodyType
}
```

The output would be the following:

```json
{
  "Planet": [
    {
      "name": "Earth",
      "type": "Planet"
    },
    {
      "name": "Mars",
      "type": "Planet"
    },
    {
      "name": "Jupiter",
      "type": "Planet"
    }
  ],
  "Moon": [
    {
      "name": "Luna",
      "type": "Moon"
    }
  ]
}
```

> [!NOTE]
> **Note:** Because each element in the `Planets` and `Moons` arrays is an object, you can add `.name` inside the function to get its 'name' field. Alternatively, you can keep the function as is and add `.name` to the function's input, like this: `root.Planets[0].name`.

## Arrays

This sections describes the different operations with arrays.

### Iteration

The syntax for iterating over an array is to add a `[]` suffix. You can use this
syntax in the following ways:

- `Function(a[])` means "pass each element of `a` (one at a time) to `Function`".
- `Function(a[], b)` means "pass each element of `a` (one at a time), along with `b` to `Function`". If `b` is an array of the same length as `a`, you can iterate over them together.
- `Function(a[], b[])` means "pass each element of `a` (one at a time), along with each element of `b` (at the same index) to `Function`".
- `[]` is also allowed after function calls: `Function2(Function(a)[])` means "pass each element from the result of `Function(a)` (one at a time) to `Function2`".
- The result of an iterating function call is also an array.

See the following example to iterative over the `Planets` and `Moons` arrays:

```python
Planet: BodyName_BodyType_BodyInfo($root.Planets[], "Planet");
Moon: BodyName_BodyType_BodyInfo($root.Moons[], "Moon");

def BodyName_BodyType_BodyInfo(body, bodyType) {
  name: body.name
  type: bodyType
}
```

With the previous input, the output would be the following:

```json
{
  "Planet": [
    {
      "name": "Earth",
      "type": "Planet"
    },
    {
      "name": "Mars",
      "type": "Planet"
    },
    {
      "name": "Jupiter",
      "type": "Planet"
    }
  ],
  "Moon": [
    {
      "name": "Luna",
      "type": "Moon"
    }
  ]
}
```

### Appending to arrays

You can append to an array using `[]` in the middle of the path. For example,
`types[].typeName: ...` is valid and creates `types: [{"typeName": ... }]`. You
can also use hardcoded indexes, such as `types[0]: ...`. "Out of bounds"
indexes, such as `types[153]: ...`, generate missing elements as `null`.

Example using index numbers:

```python
Planet[0]: "Earth"
Planet[1]: "Mars"
Planet[2]: "Jupiter"
Moon[0]: "Luna"
```

Example using appending:

```python
Planet[]: "Earth"
Planet[]: "Mars"
Planet[]: "Jupiter"
Moon[]: "Luna"
```

### Wildcards

The `[*]` syntax works the same as specifying an index, except that it returns
an array of values. Multiple arrays mapped through with `[*]`, for example
`a[*].b.c[*].d`, result in one long, non-nested array of the values of `d` with
the same item order. Hardcoded array indexes are preserved.

Null values are included through jagged traversal. For example, `a[*].b.c[*].d`.
If some instance of `a` does not contain `b.c`, then a single null value is
returned for that instance.

For example:

```python
PlanetNames: $root.Planets[*].name;
```

With the previous example, the output is the following:

```json
{
  "PlanetNames": ["Earth", "Mars", "Jupiter"]
}
```

Prepend the words "Celestial Body" to the names in `PlanetNames` using what you
learned about iterating over arrays:

```python
PlanetNames: AddPrefix("Celestial Body ", $root.Planets[]);

def AddPrefix(prefix, planet) {
  prefix + planet.name
}
```

With the previous example, the output would be the following:

```json
{
  "PlanetNames": [
    "Celestial Body Earth",
    "Celestial Body Mars",
    "Celestial Body Jupiter"
  ]
}
```

### Writing to array fields

The following steps show how to write to array fields:

```python
PlanetNames: ($root.Planets[*].name[]);

Planet: BodyName_BodyType_BodyInfo($root.Planets[], "Planet");
Moon: BodyName_BodyType_BodyInfo($root.Moons[], "Moon");

def BodyName_BodyType_BodyInfo(body, bodyType) {
  name: body.name
  types[]: bodyType
  types[]: "Body"
}
```

With the previous input, the output would be the following:

```json
{
  "Moon": [
    {
      "name": "Luna",
      "types": ["Moon", "Body"]
    }
  ],
  "Planet": [
    {
      "name": "Earth",
      "types": ["Planet", "Body"]
    },
    {
      "name": "Mars",
      "types": ["Planet", "Body"]
    },
    {
      "name": "Jupiter",
      "types": ["Planet", "Body"]
    }
  ],
  "PlanetNames": ["Earth", "Mars", "Jupiter"]
}
```

## Variables

Variables allow mapped data to be reused without re-executing it. The `var`
keyword indicates that the target field is a variable. Variables have identical
semantics to fields. You can write to or iterate over variables in the same way as any
input, however, variables don't show up in the mapping output.

> [!NOTE]
> **Note:** A variable can't have the same name as any input in its function.

The following mapping is equivalent to the mapping in
[Writing to array fields](https://docs.cloud.google.com/manufacturing-data-engine/docs/reference/whistle/basics.md.txt#writing_to_array_fields).

```python
PlanetNames: $root.Planets[*].name;

Planet: BodyName_BodyType_BodyInfo($root.Planets[], "Planet")
Moon: BodyName_BodyType_BodyInfo($root.Moons[], "Moon")

def BodyName_BodyType_BodyInfo(body, bodyType) {
  var tempName: body.name;
  name: tempName;
  types[]: bodyType;
  types[]: "Body";
}
```

## Whistle containers

A *Whistle container* is an object used to map a set of fields to values.
Containers are equivalent to a [Python
map](https://docs.python.org/3/glossary.html#term-mapping) or a JSON object
containing keys and fields.

The following snippet shows how to declare a container variable:

    var PlanetContainer: {
      Planet: "Earth"
      Moon: "Luna"
      Size: "Medium"
      Neighbors: ["Mars", "Jupiter"]
    }

## Conditions

Conditions are values that are only evaluated if a condition is met. Conditions
in Whistle are expressed as
[ternary expressions](https://docs.python.org/3/reference/expressions.html#conditional-expressions).

### Preparation

In this section, you will use planetary data taken from the
[NASA factsheets](https://nssdc.gsfc.nasa.gov/planetary/factsheet/).

```json
{
  "Planets": [
    {
      "name": "Earth",
      "semiMajorAxis": 149.6
    },
    {
      "name": "Mars",
      "semiMajorAxis": 227.92
    },
    {
      "name": "Jupiter",
      "semiMajorAxis": 778.57
    }
  ],
  "Moons": [
    {
      "name": "Luna",
      "semiMajorAxis": 0.3844
    }
  ]
}
```

Transforming the data into
[Astronomical Units (AU)](https://en.wikipedia.org/wiki/Astronomical_unit)
involves converting from the input in millions of km, using the conversion
factor 149.598 million km = 1 AU. To get the distance in AU, use the `$Div`
predefined function to divide the distance in millions of km by the conversion
constant.

```python
PlanetNames: $root.Planets[*].name;

Planet: BodyName_BodyType_BodyInfo($root.Planets[], "Planet");
Moon: BodyName_BodyType_BodyInfo($root.Moons[], "Moon");

def BodyName_BodyType_BodyInfo(body, bodyType) {
  name: body.name;
  types[]: bodyType;
  types[]: "Body";

  semiMajorAxisAU: body.semiMajorAxis / 149.598;
}
```

### Conditional mappings

The following sample demonstrates conditional mappings, which are only evaluated
if a condition is met.

Add a condition so that the `semiMajorAxisAU` field is output on planets and not
moons:

- Use the `==` (equal) operator for comparison.
- Use the `if ... then ... else ...` statement to conditionally execute the mapping.
  - The expression in the `if` statement is evaluated and the value after `then` is evaluated and returned *if and only if* the conditions are true. Otherwise, the value after `else` is evaluated and returned.
  - The `else ...` is optional and defaults to `else {}`, meaning that it returns a null value.

```python
PlanetNames: $root.Planets[*].name[];

Planet: BodyName_BodyType_BodyInfo($root.Planets[], "Planet")
Moon: BodyName_BodyType_BodyInfo($root.Moons[], "Moon")

def BodyName_BodyType_BodyInfo(body, bodyType) {
  var bigName: body.name
  name: bigName
  types[]: bodyType
  types[]: "Body"

  semiMajorAxisAU: if bodyType == "Planet" then body.semiMajorAxis / 149.598
}

```

In this example, the `/` operator divides the million km distance by the
conversion constant to produce the distance in AU.

The output would be the following:

```json
{
  "Moon": [
    {
      "name": "Luna",
      "types": ["Moon", "Body"]
    }
  ],
  "Planet": [
    {
      "name": "Earth",
      "semiMajorAxisAU": 1.0000142656266688,
      "types": ["Planet", "Body"]
    },
    {
      "name": "Mars",
      "semiMajorAxisAU": 1.5235511458665132,
      "types": ["Planet", "Body"]
    },
    {
      "name": "Jupiter",
      "semiMajorAxisAU": 5.2044191630277785,
      "types": ["Planet", "Body"]
    }
  ],
  "PlanetNames": ["Jupiter"]
}
```

### Conditional blocks

The following sample demonstrates conditional blocks, which wrap a set of
mappings within a condition. Set the `semiMajorAxis.unit` to `AU` if the
`bodyType` is a `Planet`:

```python
PlanetNames: $root.Planets[*].name[];

Planet: BodyName_BodyType_BodyInfo($root.Planets[], "Planet")
Moon: BodyName_BodyType_BodyInfo($root.Moons[], "Moon")

def BodyName_BodyType_BodyInfo(body, bodyType) {
  var bigName: body.name
  name: bigName
  types[]: bodyType
  types[]: "Body"
  if (bodyType == "Planet") then {
    semiMajorAxis.value: body.semiMajorAxis / 149.598
    semiMajorAxis.unit: "AU"
  } else {
    semiMajorAxis.value: body.semiMajorAxis * 1000000
    semiMajorAxis.unit: "KM"
  }
}
```

With the previous example, the output would be the following:

```json
{
  "Moon": [
    {
      "name": "Luna",
      "semiMajorAxis": {
        "unit": "KM",
        "value": 384400
      },
      "types": ["Moon", "Body"]
    }
  ],
  "Planet": [
    {
      "name": "Earth",
      "semiMajorAxis": {
        "unit": "AU",
        "value": 1.0000142656266688
      },
      "types": ["Planet", "Body"]
    },
    {
      "name": "Mars",
      "semiMajorAxis": {
        "unit": "AU",
        "value": 1.5235511458665132
      },
      "types": ["Planet", "Body"]
    },
    {
      "name": "Jupiter",
      "semiMajorAxis": {
        "unit": "AU",
        "value": 5.2044191630277785
      },
      "types": ["Planet", "Body"]
    }
  ],
  "PlanetNames": ["Jupiter"]
}
```

### Operators

Similar to Python or C, there are operators available for common arithmetic and
logical operations. This section shows some of the following operators,
where `num` is a number input, `bool` is a boolean input, and `any` is any type
of input:

| Operator | Description |
|---|---|
| num + num | Addition |
| num - num | Subtraction |
| num \* num | Multiplication |
| num / num | Division |
| str + any | Concatenation |
| any + str | Concatenation |
| bool and bool | Logical AND |
| bool or bool | Logical OR |
| !bool | Logical NOT |
| any == any | Deep equals - all elements in an array or values in an object must be the same to return true. `x = y = z` is a valid expression and is equivalent to `(x = y) = z`. If `x = y` is true, this checks `true = z`. |
| any != any | Not Equal |
| any? | Value exists - is defined, isn't literal `null`, and isn't empty. An empty array has 0 elements (`null`s count as elements). An empty object has 0 keys. |
| !any? | Value Does Not Exist |

## Filters

Filters have the following properties:

- Filters allow narrowing an array to items that match a condition.
- The `where` keyword indicates a filter, similar to `if` indicating a condition.
- Each item from the array is loaded one at a time into an input named `$` in the filter.
- The filter produces a new array. To iterate over the results, use the `[]` operator.
- Filters can only be the last element in a path. For example, `a.b[where
  $.color = "red"].c` is invalid.

The following steps show how to use filters with the planets example:

```python
Planet: BodyName_BodyType_BodyInfo($root.Planets[where $.semiMajorAxis > 200][], "Planet");
```

With the previous example, the output would be the following:

```json
{
  "Moon": [
    {
      "name": "Luna",
      "semiMajorAxis": {
        "unit": "KM",
        "value": 384400
      },
      "types": ["Moon", "Body"]
    }
  ],
  "Planet": [
    {
      "name": "Mars",
      "semiMajorAxis": {
        "unit": "AU",
        "value": 1.5235511458665132
      },
      "types": ["Planet", "Body"]
    },
    {
      "name": "Jupiter",
      "semiMajorAxis": {
        "unit": "AU",
        "value": 5.2044191630277785
      },
      "types": ["Planet", "Body"]
    }
  ],
  "PlanetNames": ["Jupiter"]
}
```

## Caveats

This section describes the null propagation.

### Nulls and null propagation

By default, the mapping engine handles and ignores null and missing values or
fields, according to the following rules:

- If a field is written with a null or empty value, it's ignored (thus `null`, `{}`, and `[]` can never show up in the mapping output).
- If a non-existent field is accessed, it returns `null`.
- If a null value is passed to a function, the function is still executed.

The following sample input and mapping demonstrate nulls and null propagation:

**Input**

```json
{
  "Red": {
    "Blue": 1
  }
}
```

**Mapping**

```python
def Root_Example(rt) {
  // This field does not appear in the output
  excluded: rt.Abcdefghijklmnop

  // This array will only contain the existing items
  included[]: rt.Red.Blue
  included[]: rt.Abcd[123].efghi[*].jk[*].lmnop
  included[]: rt.Red.Blue

  // nested_1 will appear with just the constant, nested_2 won't appear
  nested_1: Nested_Example(rt.Abcdefghijklmnop, "Constant")
  nested_2: Nested_Example(rt.Abcdefghijklmnop, rt.Abcdefghijklmnop)
}

def Nested_Example(one, two) {
  one: one
  two: two
}
```

The output would be similar to the following sample:

```json
{
  "Example": [
      "included": [
        1,
        1
      ],
      "nested_1": {
        "two": "Constant"
      }
  ]
}
```

### Using `root` in a function

The following sample demonstrates using the `root` keyword inside a function to
send data to the root of the output:

```python
Red[]: "Blue"
Complex: Hello_World_HelloWorldObject("Hi", "Planet")

def Hello_World_HelloWorldObject(hello, world) {
    hello: hello
    world: world
    root Red[]: world
    root Complex.boo: "boo!"
}
```

With the previous example, the output would be the following:

```json
{
  "Complex": [
    {
      "boo": "boo!",
      "hello": "Hi",
      "world": "Planet"
    }
  ],
  "Red": ["Blue", "Planet"]
}
```