# Xan zine: March 2024 Edition

Upgrade to the latest version (at least 0.6.0) using:

```bash
cargo install xan
```

## Summary

* [Dedup command](#dedup-command)
* [Transpose command](#transpose-command)
* [Range command](#range-command)
* [From command](#from-command)
* [Full-fledged pipeline operator](#full-fledged-pipeline-operator)
* [Column selection for groupby](#column-selection-for-groupby)
* [Stats and frequency commands overhaul](#stats-and-frequency-commands-overhaul)
* [Select using an expression](#select-using-an-expression)
* [New aggregation functions](#new-aggregation-functions)
* [Dropping moonblade special identifiers](#dropping-moonblade-special-identifiers)
* [Sorted partition optimization](#sorted-partition-optimization)
* [Failed parallelization attempts](#failed-parallelization-attempts)
* [The all-encompassing star](#the-all-encompassing-star)
* [Join support for streams and gzip](#join-support-for-streams-and-gzip)
* [Multiple explosion and implosion 🔥😎🔥](#multiple-explosion-and-implosion-🔥😎🔥)
* [Bytesize moonblade function](#bytesize-moonblade-function)

## Dedup command

Yes, it is already possible to deduplicate a file by using `xan sort -u`. But it has three downsides:

1. it needs to buffer the whole file into memory
2. it has `O(n log n)` time complexity
3. it will affect the order of lines, as a side effect

The third point can of course be mitigated by using `xan enum` before `xan sort -u` and then reordering on the `enum` column afterwards. But this is quite inefficient and convoluted.

So let's introduce the `dedup` command. It runs in amortized `O(n)` time by storing row identities (based on a given column selection) in a hashset, and only consumes `O(c)` memory, where `c` is the number of distinct row identities.

The command is also able to leverage the fact that you know the file is already sorted on the identity columns (using the `-S/--sorted` flag), to only require constant memory.

Note that first row will always be favored, against subsequent ones having the same identity. This can be important when not deduplicating on all the columns.

```bash
xan dedup file.csv

# Deduplicate, based on a single column
xan dedup -s name file.csv

# Deduplicate a sorted file
xan dedup --sorted -s name file.csv
```

*Dedicated to @pournaki*

## Transpose command

The new `transpose` command can be used to transpose a CSV file. That's it.

| A | B | C |
|---|---|---|
| 1 | 2 | 3 |

Would become:

| A | 1 |
|---|---|
| B | 2 |
| C | 3 |

I personally use it to recombobulate `xan stats` results piped into `xan view` sometimes, for readability.

## Range command

This new command is able to generate a CSV file with a single column representing a numerical range:

```bash
xan range 3
```

| n |
|---|
| 0 |
| 1 |
| 2 |

This one might seem pointless, but it is a good idea to pipe its results into other commands such as `transform` or `map` to generate more complex files.

For instance, let's imagine we want to download [https://www.echojs.com](https://www.echojs.com) pages. If we go to their "latest" page: [https://www.echojs.com/latest/0](https://www.echojs.com/latest/0) and click on "more" at the bottom, we reach [https://www.echojs.com/latest/30](https://www.echojs.com/latest/30).

So one way to do this could be to generate a CSV file of urls to feed to `minet fetch` to efficiently download them:

```bash
xan range 10 | xan map '"https://www.echojs.com/latest/" . (n * 30)' url | minet fetch url -i -
```

Finally, note that this command supports the usual `range` tricks you may find in any programming language: custom starting point, custom step, etc.

## From command

The `jsonl` and `xls` commands have been dropped. But they have been replaced by a new command named `from`, that is able to consume various data formats and output them as CSV:

```bash
# Converting newline-delimited JSON
xan from data.ndjson

# Converting from a stream, we cannot infer the format, so we must provide it
# to the command using -f
cat data.ndjson | xan from -f ndjson

# Converting Excel files
xan from data.xlsx
```

What's more, `ndjson` conversion has been vastly improved. It should now be more performant and more precise, as it actually samples some of the first lines to make sure we can get the union of keys and paths from more than the first record. This way, we can flatten the data hierarchy into neat columns, sorted by type (scalars coming first), depth and key.

This also means we can imagine some kind of `to` command in the future.

## Full-fledged pipeline operator

Until now, only some commands were able to use the weird pipeline variant of the `moonblade` expression language.

It was looking like this:

```bash
xan map 'read(path) | trim(_) | len'
```

But now, `moonblade` has a full-fledged pipeline operator `|` akin to what can be found in the [F#](https://en.wikipedia.org/wiki/F_Sharp_(programming_language)) or the [Hack](https://hacklang.org/) programming languages.

It can be used anywhere in an expression as a low-precedence operator and accepts
`_` as argument substitution (e.g. in `trim(_)`, `_` represent earlier computation) and tolerates argument elision for unary functions (e.g. `| len` is a shorthand for `| len(_)`).

This means the following expression is now perfectly valid xan:

```javascript
add(read(path) | trim(_) | len, 64)
```

## Column selection for groupby

Until now, the `groupby` command was only able to use a single column as group identity. But now, the command will work on any column selection.

Furthermore, the command used to return a `group` column containing the grouped values. It will now keep the original names from the selected columns instead.

So, given this very interesting CSV file:

| name | surname |
|------|---------|
| John | Wick    |
| Mary | Sue     |
| John | Wick    |

The following command:

```bash
xan groupby name,surname 'count() as count' file.csv
```

Will produce:

| name | surname | count |
|------|---------|-------|
| John | Wick    | 2     |
| Mary | Sue     | 2     |


This means we can now run custom aggregations on combinations of columns very easily. It is still impossible for the `frequency` command though, because its selection will build a frequency table per column instead.

## Stats and frequency commands overhaul

The `stats` and `frequency` commands have been completely rewritten. They should be faster and cleaner now.

Some unused `frequency` flags have been removed, and the command is now able to leverage heaps to be a little bit faster and avoid allocating unnecessary memory.

The `stats` command still compute constant-memory descriptive statistics by default and now splits additional metrics in `--quartiles` and `--cardinality`.

Here is the full list of statistics produced by the command:

```
field         (default) - Name of the described column
count         (default) - Number of non-empty values contained by the column
type          (default) - Most likely type of the column
types         (default) - Pipe-separated list of all types witnessed in the column
sum           (default) - Sum of numerical values
mean          (default) - Mean of numerical values
q1            (-q)      - First quartile of numerical values
median        (-q)      - Second quartile, i.e. median, of numerical values
q3            (-q)      - Third quartile of numerical values
variance      (default) - Population variance of numerical values
stddev        (default) - Population standard deviation of numerical values
min           (default) - Minimum numerical value
max           (default) - Maximum numerical value
cardinality   (-c)      - Number of distinct string values
mode          (-c)      - Most frequent string value (tie breaking is arbitrary & random!)
tied_for_mode (-c)      - Number of values tied for mode
lex_first     (default) - First string in lexical order
lex_last      (default) - Last string in lexical order
min_length    (default) - Minimum string length
max_length    (default) - Maximum string length
```

## Select using an expression

It is now possible to use a `moonblade` expression to select columns in a SQLish manner, using the `-e/--evaluate` flag like so:

```bash
xan select -e 'name as first_name, last_name, earnings + savings as total' file.csv
```

No. Limit.

## New aggregation functions

*Quantiles*

Now, beyond of median, it is now possible to compute arbitrary quantiles using the `quantile` aggregator.

`q1` and `q3` utility aggregators have also been added for convenience to retrieve the first and third quartiles.

Note that quantile computation methods are actually quite a divisive topic in itself. Especially if you want to compute a one-shot quantile, vs. reporting all of them fitting a specific partition of the dataset.

This means the `quantile` function return an interpolated one-shot version of what the desired quantile would be, whereas the `q1`, `median` and `q3` functions return quartiles that are consistently dividing the data.

*Argmin, Argmax*

The `argmin/argmax` aggregator returns a value associated with another value's minimum or maximum. It can also return the index of the row if the associated value is not specified.

```js
// Returns the index of the row with min `number` column
argmin(number)

// Basically same as:
argmin(number, index())

// Returns the value of the `name` column where the `number` column is minimized
argmin(number, name)
```

*Type guessing*

The `type` aggregator returns the most probable type for a group of values (most generic types being favored. e.g. if we saw some ints and floats, we will infer the values to be floats).

Currently detected types are: `empty`, `int`, `float` & `string`.

The `types` aggregator, on the other hand, returns a sorted list, separated by `|`, of all types witnessed in a group of values. This is quite useful to analyze columns containing potentially mixed types.

*Values reporting*

The `values` aggregator returns all seen values, separated by a provided character (`|`, by default).

The `distinct_values` aggregator return all distinct & sorted values, separated by a provided character (`|`, by default).

*Modes*

The `modes` aggregator return all values tied for mode, separated by a provided character (`|`, by default).

## Dropping moonblade special identifiers

The `moonblade` expression language used to have a special identifier syntax, prefixed with `%` that could be used to access special values such as the current row's index through `%index`.

This tidbit has been dropped to simplify the language and to get even closer to a SQLish expression syntax.

It does not mean you cannot access the current row's index anymore, but now you will need to use the `index()` function, like with most SQL implementations.

```bash
xan map 'index()' index file.csv
# Very close to:
xan enum file.csv
```

## Sorted partition optimization

Like other `xan` commands, `partition` can now take a `-S/--sorted` flag when you know the file is already sorted on the partition column.

This makes the command both faster and able to work with less memory and resources. This also circumvents the limitation about the maximum number of files your OS is able to keep opened at once.

## Failed parallelization attempts

I tried very hard to parallelize some commands such as `stats`, `frequency`, `agg` & `groupby`. But I must admit I cannot find a way to make it worthwhile most of the time.

I tried 5 different things:

*1. Horizontal parallelization*

Let's say we want to run the following aggregation:

```bash
xan agg 'mean(A), sum(B)' file.csv
```

There are two distinct computations to be done, on two different columns of the input data. We can do them in parallel.

*2. Parallelized finalization*

Some aggregations, the `median` for instance, require to store all seen numbers, then to sort them. We can parallelize the finalization of those aggregations, in this case the sorting can be done in parallel.

*3. Vertical parallelization*

Here we read the rows of the CSV file and we send them to a pool of threads. This means each thread work on its own version of the aggregation (stored in the thread's local storage), and we must merge all those versions once the work is done.

This puts some pressure on the aggregator's implementation because you need to make sure they can be merged after the fact. This is usually doable, but not always.

*4. Vertical parallelization, chunked*

Same as above, but instead of sending rows one by one to our threads, we send larger chunk of them, to offset the synchronization costs and benefit from the fact that reading our file is usually done in a buffered manner.

*5. Concurrent hashmaps*

This mostly relates to the `groupby` & `frequency` commands. Here, instead of having one hashmap per thread that we need to merge after the fact, and instead of using a hashmap behind a lock, we use a hashmap designed for concurrency, such as [`dashmap`](https://docs.rs/dashmap/latest/dashmap/).

*Conclusion*

Each time, the overhead of sending data around CPUs defeated the purpose and made the parallelized code infuriatingly slower or as performant as the single-threaded version.

The outline here is that the cost of reading & buffering the CSV file actually dwarfs the operations we are running on the rows themselves. This means parallelization in this context is just raw overhead without any advantage.

This said, it also means that if the computations you have to run per row are costly, i.e. matching a complex regular expression or reading a file on disk, parallelization **will** provide a boost in performance.

```bash
# This will not be faster with parallelization
xan agg -p 'sum(if(col eq "yes", 1, 0))'

# This will absolutely be faster
xan agg -p 'sum(if("logged" in read(path), 1, 0))'
```

## The all-encompassing star

The selection mini DSL now accepts `*` to mean all columns. This can be useful sometimes, e.g. to add a copy of some column (SQL-style baby) using the `select` command:

```bash
# We will have 2 "name" columns
xan select '*,name' file.csv
```

## Join support for streams and gzip

The `join` command is now able to work with `stdin` (using the `-` shorthand, since we cannot chose for you if stdin must be the left or right file) and gzipped files.

I have still some work to do to optimize this because right now this works by buffering those files into memory. Usually, only one of both files provided to the command actually requires to be fully buffered.

## Multiple explosion and implosion 🔥😎🔥

The `explode` and `implode` commands used to work on a single column. But fact is we often deal with files that have multiple columns that are separated by some character (often `|`), containing aligned information.

This is the case, for instance, with our CSV files containing tweets where one column stores the media urls and another one the media types.

So now, given this file:

| name | colors     | letters |
|------|------------|---------|
| John | red\|yellow | A\|B     |

We will be able to produce:

| name | color  | letter |
|------|--------|--------|
| John | red    | A      |
| John | yellow | B      |

using the following command:

```bash
xan explode colors,letters '|' -r color,letter file.csv
```

And the reverse is also true, using the `implode` command.

## Bytesize moonblade function

This handy new `moonblade` function added by bmaz makes it easy to convert an arbitrary number of bytes into a human-readable number like `10G`.

Here is an example where I compute the sum of the size of files downloaded by a crawler and make it readable using the new `bytesize` function:

```bash
xan agg 'sum(coalesce(body_size, 0)) as total' jobs.csv| xan transform total bytesize
```
