arquero

Arquero API Reference

Top-Level Table Verbs Op Functions Expressions Extensibility


Op Functions

Add new functions for use in table expressions.


# aq.addFunction([name,] fn[, options]) · Source

Register a function for use within table expressions. If only a single argument is provided, it will be assumed to be a function and the system will try to extract its name. Throws an error if a function with the same name is already registered and the override option is not specified, or if no name is provided and the input function is anonymous. After registration, the function will be accessible via the op object.

Also see the escape() expression helper for a lightweight alternative that allows access to functions defined in an enclosing scope.

Examples

// add a function named square, which is then available as op.square()
// if a 'square' function already exists, this results in an error
aq.addFunction('square', x => x * x);
// add a function named square, override any previous 'square' definition
aq.addFunction('square', x => x * x, { override: true });
// add a function using its existing name
aq.addFunction(function square(x) { return x * x; });

# aq.addAggregateFunction(name, def[, options]) · Source

Register a custom aggregate function. Throws an error if a function with the same name is already registered and the override option is not specified. After registration, the operator will be accessible via the op object.

In addition to column values, internally aggregate functions are passed a state object that tracks intermediate values throughout the aggregation. Each groupby group receives a different state object. The state object always has a count property (total number of values, including invalid values) and a valid property (number of values that are not null, undefined, or NaN). Each aggregate operator may write intermediate values to the state object. Follow the property naming convention of using the aggregate function name as a property name prefix to avoid namespace collisions! For example, the mean aggregate function writes to the properties state.mean and state.mean_d.

The rem function of an aggregate definition is used to support rolling window calculations. It is safe to define rem as a no-op (() => {}) if the aggregate is never used in the context of a rolling window frame.

Examples

// add an aggregate function that computes a sum of squares
// the "rem" method is only needed for windowed aggregates
aq.addAggregateFunction('sumsq', {
  create: () => ({
    init:  state => state.sumsq = 0,
    add:  (state, value) => state.sumsq += value * value,
    rem:  (state, value) => state.sumsq -= value * value,
    value: state => state.sumsq
  }),
  param: [1, 0] // 1 field input, 0 extra parameters
});

aq.table({ x: [1, 2, 3] })
  .rollup({ ssq: op.sumsq('x') } // { ssq: [14] }

# aq.addWindowFunction(name, def[, options]) · Source

Register a custom window function. Throws an error if a function with the same name is already registered and the override option is not specified. After registration, the operator will be accessible via the op object.

Examples

// add a window function that outputs the minimum of a field value
// and the current sorted window index, plus an optional offset
aq.addWindowFunction('idxmin', {
  create: (offset = 0) => ({
    init: () => {}, // do nothing
    value: (w, f) => Math.min(w.value(w.index, f), w.index) + offset
  }),
  param: [1, 1] // 1 field input, 1 extra parameter
});

aq.table({ x: [4, 3, 2, 1] })
  .derive({ x: op.idxmin('x', 1) }) // { x: [1, 2, 3, 2] }


Table Methods

Add new table-level methods or verbs. The addTableMethod function registers a new function as an instance method of tables only. The addVerb method registers a new transformation verb with both tables and serializable queries.


# aq.addTableMethod(name, method[, options]) · Source

Register a custom table method, adding a new method with the given name to all table instances. The provided method must take a table as its first argument, followed by any additional arguments.

This method throws an error if the name argument is not a legal string value. To protect Arquero internals, the name can not start with an underscore (_) character. If a custom method with the same name is already registered, the override option must be specified to overwrite it. In no case may a built-in method be overridden.

Examples

// add a table method named size, returning an array of row and column counts
aq.addTableMethod('size', table => [table.numRows(), table.numCols()]);
aq.table({ a: [1,2,3], b: [4,5,6] }).size() // [3, 2]

# aq.addVerb(name, method, params[, options]) · Source

Register a custom transformation verb with the given name, adding both a table method and serializable query support. The provided method must take a table as its first argument, followed by any additional arguments. The required params argument describes the parameters the verb accepts. If you wish to add a verb to tables but do not require query serialization support, use addTableMethod.

This method throws an error if the name argument is not a legal string value. To protect Arquero internals, the name can not start with an underscore (_) character. If a custom method with the same name is already registered, the override option must be specified to overwrite it. In no case may a built-in method be overridden.

Parameter Types. The supported parameter types are:

Examples

// add a bootstrapped confidence interval verb that
// accepts an aggregate expression plus options
aq.addVerb(
  'bootstrap_ci',
  (table, expr, options = {}) => table
    .params({ frac: options.frac || 1000 })
    .sample((d, $) => op.round($.frac * op.count()), { replace: true })
    .derive({ id: (d, $) => op.row_number() % $.frac })
    .groupby('id')
    .rollup({ bs: expr })
    .rollup({
      lo: op.quantile('bs', options.lo || 0.025),
      hi: op.quantile('bs', options.hi || 0.975)
    }),
  [
    { name: 'expr', type: 'Expr' },
    { name: 'options', type: 'Options' }
  ]
);

// apply the new verb
aq.table({ x: [1, 2, 3, 4, 6, 8, 9, 10] })
  .bootstrap_ci(op.mean('x'))


Package Bundles

Extend Arquero with a bundle of functions, table methods, and/or verbs.


# aq.addPackage(bundle[, options]) · Source

Register a bundle of extensions, which may include standard functions, aggregate functions, window functions, table methods, and verbs. If the input bundle has a key named "arquero_package", the value of that property is used; otherwise the bundle object is used directly. This method is particularly useful for publishing separate packages of Arquero extensions and then installing them with a single method call.

A package bundle has the following structure:

const bundle = {
  functions: { ... },
  aggregateFunctions: { ... },
  windowFunctions: { ... },
  tableMethods: { ... },
  verbs: { ... }
};

All keys are optional. For example, functions or verbs may be omitted. Each sub-bundle is an object of key-value pairs, where the key is the name of the function and the value is the function to add.

The lone exception is the verbs bundle, which instead uses an object format with method and params keys, corresponding to the method and params arguments of addVerb:

const bundle = {
  verbs: {
    name: {
      method: (table, expr) => { ... },
      params: [ { name: 'expr': type: 'Expr' } ]
    }
  }
};

The package method performs validation prior to adding any package content. The method will throw an error if any of the package items fail validation. See the addFunction, addAggregateFunction, addWindowFunction, addTableMethod, and addVerb methods for specific validation criteria. The options argument can be used to specify if method overriding is permitted, as supported by each of the aforementioned methods.

Examples

// add a package
aq.addPackage({
  functions: {
    square: x => x * x,
  },
  tableMethods: {
    size: table => [table.numRows(), table.numCols()]
  }
});
// add a package, ignores any content outside of "arquero_package"
aq.addPackage({
  arquero_package: {
    functions: {
      square: x => x * x,
    },
    tableMethods: {
      size: table => [table.numRows(), table.numCols()]
    }
  }
});
// add a package from a separate library
aq.addPackage(require('arquero-arrow'));