> ## Documentation Index
> Fetch the complete documentation index at: https://docs.callivox.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Working with data

> Reshape lists inside a flow: map fields, filter rows, group and aggregate, deduplicate, sort, build arrays, convert JSON — and run a JavaScript snippet when nothing else fits.

Most flows start by fetching a list — rows from a sheet, records from a CRM, items from an API — and
the shape you get is rarely the shape you want. The **Data** section of the palette reshapes it
before it reaches whatever comes next.

Every Data block takes a list in and produces **Result** (the new list) plus **Count**, so they chain
end to end. A list can be a previous step's output, JSON text, or a single object.

## The blocks

<CardGroup cols={2}>
  <Card title="Map Items" icon="shuffle">
    Keep only the fields you name, under the names you want. The biggest lever on payload size.
  </Card>

  <Card title="Filter Items" icon="filter">
    Keep only the items matching a condition, with an optional second condition.
  </Card>

  <Card title="Group & Aggregate" icon="layer-group">
    Collapse many rows into one item per group — count them, total them, merge their values.
  </Card>

  <Card title="Unique Items" icon="copy">
    Remove repeats, keeping the first of each.
  </Card>

  <Card title="Sort Items" icon="arrow-up-a-z">
    Order a list by a field, ascending or descending.
  </Card>

  <Card title="Take Items" icon="scissors">
    Take the first N items, optionally after skipping some.
  </Card>

  <Card title="Build Array" icon="plus">
    Create a list from scratch, or append to one you already have.
  </Card>

  <Card title="Parse JSON" icon="brackets-curly">
    Turn JSON text into a list you can map, filter and group.
  </Card>

  <Card title="To JSON Text" icon="file-code">
    Turn a list back into JSON text for a response or message body.
  </Card>

  <Card title="Run JavaScript" icon="code">
    A short script, for anything the blocks above cannot express.
  </Card>
</CardGroup>

Field names are matched **without regard to capitalisation**, and a name with dots in it reaches
inside nested values — `customer.city` reads the `city` inside `customer`.

## Map Items

Give it a **Field mapping**: a JSON object of the name you want on the left, the field it comes from
on the right.

```json theme={null}
{ "name": "Doctor Name", "department": "Department", "languages": "Languages" }
```

Everything not listed is dropped. Turn on **Keep unmapped fields** to keep the rest as well, in which
case a mapping simply renames.

## Filter Items

Pick a **Field**, a **Condition** and a **Value**. Available conditions: equals, does not equal,
contains, does not contain, starts with, ends with, greater than, greater than or equal, less than,
less than or equal, is empty, is not empty, is one of, is not one of. The last two take a
comma-separated list.

Add a **Second field** to test two things at once, combined with **AND** (both must match) or **OR**
(either may match).

<Note>
  Comparisons like *greater than* compare **numbers as numbers** when both sides look numeric, so
  `10` is correctly greater than `9`. You do not have to declare a column type.
</Note>

## Group & Aggregate

This is the block for turning a repeated export into one row per thing. A schedule with one row per
doctor per day becomes one row per doctor.

Set **Group by** to the field (or comma-separated fields) that identify a group, then list what each
group should contain in **Aggregations**:

```json theme={null}
[
  { "name": "department", "op": "first",         "field": "Department" },
  { "name": "locations",  "op": "join_distinct", "field": "Location", "separator": ", " },
  { "name": "visits",     "op": "count" }
]
```

Each entry needs a **name** (the output field) and an **op**. Every op except `count` needs a
**field**.

| Op                            | Result                                                            |
| ----------------------------- | ----------------------------------------------------------------- |
| `count` / `count_distinct`    | How many items, or how many different values                      |
| `first` / `last`              | The first or last value in the group                              |
| `sum` / `avg` / `min` / `max` | Numeric totals and extremes                                       |
| `join` / `join_distinct`      | The values joined into one string, all or only the different ones |
| `list` / `list_distinct`      | The values as a list                                              |

**Include group fields** carries the group-by fields onto each result item. Leave it on unless you
have a reason not to.

<Warning>
  **When a field varies inside a group, `first` throws the rest away.** A doctor who works at three
  clinics has three different Locations in the group; `first` keeps one and silently loses two.
  Use `join_distinct` (or `list_distinct`) for any field that legitimately differs within a group.
  The same applies to **Unique Items** — it keeps one whole item and discards the others.
</Warning>

<Note>
  If you leave **Aggregations** empty *and* turn **Include group fields** off, there is nothing to
  put in each group. The step will tell you rather than quietly returning empty items.
</Note>

### Filter before you group

Rows that should not count must be removed *before* grouping. In a staff schedule, a day someone is
off is still a row, and it still carries a location — group without filtering and you will credit
people with places they never work.

<Steps>
  <Step title="Filter Items">
    `Time Start` **does not equal** `Off`
  </Step>

  <Step title="Group & Aggregate">
    Group by `Doctor Name`, merging `Location` with `join_distinct`
  </Step>

  <Step title="Map Items">
    Keep just the fields the answer needs
  </Step>

  <Step title="To JSON Text">
    Compact, into the response body
  </Step>
</Steps>

## Run JavaScript

For a transform the other blocks cannot express. Prefer a regular block where one fits — those show
their inputs and outputs in the run history, while a script is opaque when something goes wrong.

```js theme={null}
var rows = getOutputFrom('sheets-find-1', 'Rows');
var out = [];
for (var r of rows) {
  out.push({ name: r['Doctor Name'], dept: r.Department });
}
out;
```

* Read an earlier step with **`getOutputFrom('<step id>', '<Output>')`**. The **Use a previous step**
  picker adds the reference to the end of your script.
* Rows behave like objects: `r['Doctor Name']` and `r.Department` both work.
* **End with a plain expression** — `out;` — because the result is the value of the last expression.
  A script that ends with a loop returns that loop's last value instead, which is rarely what you
  meant.

Outputs are **Result** (the value), **Count** (items, when it is a list) and **Text** (the value as
JSON).

<Note>
  Scripts run in a sandbox. They cannot reach the server, the filesystem or the network, and they are
  limited in how long they may run and how much memory they may use — a script that never finishes
  fails its step instead of holding up anything else.
</Note>

## Feeding an AI agent tool

When a flow answers an [AI agent tool](/ai-agents/tools), what the agent receives is capped at
**24,576 characters**. Beyond that the response is cut short, and the agent is told it is incomplete
so it can ask again more narrowly — but a shortened answer is still a worse answer.

The fix is to send less, not to send more:

<CardGroup cols={2}>
  <Card title="Send only what is needed" icon="compress">
    **Map Items** to a handful of fields. Dropping the columns a question never asks about is the
    single biggest saving.
  </Card>

  <Card title="Split by question" icon="arrows-split-up-and-left">
    One tool returning a directory, another taking filters and returning detail, beats one tool
    trying to return everything.
  </Card>

  <Card title="Stay compact" icon="minimize">
    Leave **Indented** off in **To JSON Text** — formatting is wasted space for a machine reader.
  </Card>

  <Card title="Watch the size" icon="ruler">
    **To JSON Text** reports **Length**, so you can branch when a payload grows.
  </Card>
</CardGroup>

## Next steps

<CardGroup cols={2}>
  <Card title="Building a flow" icon="diagram-project" href="/automations/building-a-flow">
    The canvas, the palette, connecting steps and publishing.
  </Card>

  <Card title="Runs and troubleshooting" icon="list-check" href="/automations/runs-and-troubleshooting">
    See what a step actually produced when a flow misbehaves.
  </Card>
</CardGroup>
