Preprocess

This module contains functions for preprocessing data (filtering and standarization), the batch/ folder contains code to preprocess data using the YASS batch processor (yass.batch), experimental/ has a faster (but unstable implementation). The bottleneck in the stable implementation is due to the BatchProcessor, a re-implementation of the binary reader used there should fix the performance issues.

yass.preprocess.run(if_file_exists='skip', function=<function run>, **function_kwargs)[source]

Preprocess pipeline: filtering, standarization and whitening filter

This step (optionally) performs filtering on the data, standarizes it and computes a whitening filter. Filtering and standarized data are processed in chunks and written to disk.

Parameters:
if_file_exists: str, optional

One of ‘overwrite’, ‘abort’, ‘skip’. Control de behavior for every generated file. If ‘overwrite’ it replaces the files if any exist, if ‘abort’ it raises a ValueError exception if any file exists, if ‘skip’ it skips the operation (and loads the files) if any of them exist

function: function, optional

Which function to use, either yass.preprocess.batch.run or yass.preprocess.experimental.run. See the respective files for differences

**function_kwargs: keyword arguments

Keyword arguments passed to function. First argument is CONFIG, then if_file_exists and then this keyword parameters
Returns:
standarized_path: str

Path to standarized data binary file

standarized_params: str

Path to standarized data parameters

whiten_filter: numpy.ndarray

Whiten matrix

Notes

Running the preprocessor will generate the followiing files in CONFIG.data.root_folder/output_directory/:

  • preprocess/filtered.bin - Filtered recordings
  • preprocess/filtered.yaml - Filtered recordings metadata
  • preprocess/standarized.bin - Standarized recordings
  • preprocess/standarized.yaml - Standarized recordings metadata
  • preprocess/whitening.npy - Whitening filter

Everything is run on CPU.

Examples

import logging

import yass
from yass import preprocess

# configure logging module to get useful information
logging.basicConfig(level=logging.INFO)

# set yass configuration parameters
yass.set_config('config.yaml', 'example-preprocess')

# run preprocessor
(standarized_path,
 standarized_params,
 whiten_filter) = preprocess.run()