#message(FATAL_ERROR"Please set the path to the fluid_decomposition sources with -DFLUID_DECOMP_PATH=<path>")
#endif()
@ -46,11 +43,11 @@ if (NOT (EXISTS "${FLUID_ABS_PATH}/build/fluid_decomposition-exports.cmake"))
message(FATAL_ERROR"Can't find the fluid_decomposition CMake targets file at ${FLUID_ABS_PATH}/build/fluid_decomposition-exports.cmake. Please go to ${FLUID_ABS_PATH}/build and run CMake")
This class implements an amplitude-based slicer, with various customisable options and conditions to detect absolute and relative amplitude changes as onsets and offsets. It is part of the Fluid Decomposition Toolkit of the FluCoMa project.footnote::This was made possible thanks to the FluCoMa project ( http://www.flucoma.org/ ) funded by the European Research Council ( https://erc.europa.eu/ ) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No 725899).::
FluidAmpSlice is based on two envelop followers on a highpassed version of the signal: one absolute, and one relative. Each have features that will interact, including independent Schmidt triggers and state-aware time contraints. The example code below is unfolding the various possibilites in order of complexity.
The process will return a two-channel buffer with the addresses of the onset on the first channel, and the address of the offset on the second channel.
CLASSMETHODS::
METHOD:: process
This is the method that calls for the slicing to be calculated on a given source buffer.
ARGUMENT:: server
The server on which the buffers to be processed are allocated.
ARGUMENT:: source
The index of the buffer to use as the source material to be sliced through novelty identification. The different channels of multichannel buffers will be summed.
ARGUMENT:: startFrame
Where in the srcBuf should the slicing process start, in sample.
ARGUMENT:: numFrames
How many frames should be processed.
ARGUMENT:: startChan
For multichannel sources, which channel should be processed.
ARGUMENT:: numChans
For multichannel sources, how many channel should be summed.
ARGUMENT:: indices
The index of the buffer where the indices (in sample) of the estimated starting points of slices will be written. The first and last points are always the boundary points of the analysis.
ARGUMENT:: absRampUp
The number of samples the absolute envelope follower will take to reach the next value when raising.
ARGUMENT:: absRampDown
The number of samples the absolute envelope follower will take to reach the next value when falling.
ARGUMENT:: absThreshOn
The threshold in dB of the absolute envelope follower to trigger an onset, aka to go ON when in OFF state.
ARGUMENT:: absThreshOff
The threshold in dB of the absolute envelope follower to trigger an offset, , aka to go ON when in OFF state.
ARGUMENT:: minSliceLength
The length in samples that the Slice will stay ON. Changes of states during that period will be ignored.
ARGUMENT:: minSilenceLength
The length in samples that the Slice will stay OFF. Changes of states during that period will be ignored.
ARGUMENT:: minLengthAbove
The length in samples that the absolute envelope have to be above the threshold to consider it a valid transition to ON. The Slice will start at the first sample when the condition is met. Therefore, this affects the latency.
ARGUMENT:: minLengthBelow
The length in samples that the absolute envelope have to be below the threshold to consider it a valid transition to OFF. The Slice will end at the first sample when the condition is met. Therefore, this affects the latency.
ARGUMENT:: lookBack
The length of the buffer kept before an onset to allow the algorithm, once a new Slice is detected, to go back in time (up to that many samples) to find the minimum amplitude as the Slice onset point. This affects the latency of the algorithm.
ARGUMENT:: lookAhead
The length of the buffer kept after an offset to allow the algorithm, once the Slice is considered finished, to wait further in time (up to that many samples) to find a minimum amplitude as the Slice offset point. This affects the latency of the algorithm.
ARGUMENT:: relRampUp
The number of samples the relative envelope follower will take to reach the next value when raising. Typically, this will be faster than absRampUp.
ARGUMENT:: relRampDown
The number of samples the relative envelope follower will take to reach the next value when falling. Typically, this will be faster than absRampDown.
ARGUMENT:: relThreshOn
The threshold in dB of the relative envelope follower to trigger an onset, aka to go ON when in OFF state. It is computed on the difference between the two envelope followers.
ARGUMENT:: relThreshOff
The threshold in dB of the relative envelope follower to reset, aka to allow the differential envelop to trigger again.
ARGUMENT:: highPassFreq
The frequency of the fourth-order Linkwitz–Riley high-pass filter (https://en.wikipedia.org/wiki/Linkwitz%E2%80%93Riley_filter). This is done first on the signal to minimise low frequency intermodulation with very fast ramp lengths.
ARGUMENT:: outputType
(describe argument here)
ARGUMENT:: action
A Function to be evaluated once the offline process has finished and indices instance variables have been updated on the client side. The metric will be passed indices as an argument.
RETURNS::
Nothing, as the destination buffer is declared in the function call.
EXAMPLES::
code::
// define a test signal and a destination buffer
(
b = Buffer.sendCollection(s, Array.fill(44100,{|i| sin(i*pi/ (44100/640)) * (sin(i*pi/ 22050)).abs}));
The index of the buffer where the indices (in sample) of the estimated starting points of slices will be written. The first and last points are always the boundary points of the analysis.
ARGUMENT:: feature
The feature on which novelty is computed.
table::
##0 || Spectrum || The magnitude of the full spectrum.
The granularity of the window in which the algorithm looks for change, in samples. A small number will be sensitive to short term changes, and a large number should look for long term changes.
//check the number of slices: it is the number of frames in the transBuf minus the boundary index.
c.query;
@ -124,13 +132,13 @@ c.query;
}.play;
)
// change the filterSize in the code above to 4. Then to 8. Listen in between to the differences.
// change the filterSize in the code above to 4. Then to 12. Listen in between to the differences.
// What's happening? In the first instance (filterSize = 1), the novelty line is jittery and therefore overtriggers on the arpegiated guitar. We also can hear attacks at the end of the segment. Setting the threshold higher (like in the 'Basic Example' pane) misses some more subtle variations.
// So in the second settings (filterSize = 4), we smooth the novelty line a little, which allows us to catch small differences that are not jittery. It also corrects the ending cutting by the same trick: the averaging of the sharp pick is sliding up, crossing the threshold slightly earlier.
// If we smooth too much, like the third settings (filterSize = 8), we start to loose precision. Have fun with different values of theshold then will allow you to find the perfect segment for your signal.
// If we smooth too much, like the third settings (filterSize = 12), we start to loose precision and miss attacks. Have fun with different values of theshold then will allow you to find the perfect segment for your signal.
This class implements many spectral-based onset detection functions, most of them taken from the literature. (http://www.dafx.ca/proceedings/papers/p_133.pdf) Some are already available in SuperCollider's LINK::Classes/Onsets:: object yet not as offline processes. It is part of the Fluid Decomposition Toolkit of the FluCoMa project.footnote::This was made possible thanks to the FluCoMa project ( http://www.flucoma.org/ ) funded by the European Research Council ( https://erc.europa.eu/ ) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No 725899).::
This class implements many spectral-based onset detection metrics, most of them taken from the literature. (http://www.dafx.ca/proceedings/papers/p_133.pdf) Some are already available in SuperCollider's LINK::Classes/Onsets:: object yet not as offline processes. It is part of the Fluid Decomposition Toolkit of the FluCoMa project.footnote::This was made possible thanks to the FluCoMa project ( http://www.flucoma.org/ ) funded by the European Research Council ( https://erc.europa.eu/ ) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No 725899).::
The process will return a buffer which contains indices (in sample) of estimated starting points of different slices.
@ -34,8 +34,8 @@ ARGUMENT:: numChans
ARGUMENT:: indices
The index of the buffer where the indices (in sample) of the estimated starting points of slices will be written. The first and last points are always the boundary points of the analysis.
ARGUMENT:: function
The function used to derive a difference curve between spectral frames. It can be any of the following:
ARGUMENT:: metric
The metric used to derive a difference curve between spectral frames. It can be any of the following:
TABLE::
##0 || Energy || thresholds on (sum of squares of magnitudes / nBins) (like Onsets \power)
##1 || HFC || thresholds on (sum of (squared magnitudes * binNum) / nBins)
@ -50,7 +50,7 @@ ARGUMENT:: function
::
ARGUMENT:: threshold
The thresholding of a new slice. Value ranges are different for each function, from 0 upwards.
The thresholding of a new slice. Value ranges are different for each metric, from 0 upwards.
ARGUMENT:: minSliceLength
The minimum duration of a slice in number of hopSize.
@ -59,7 +59,7 @@ ARGUMENT:: filterSize
The size of a smoothing filter that is applied on the novelty curve. A larger filter filter size allows for cleaner cuts on very sharp changes.
ARGUMENT:: frameDelta
For certain functions (HFC, SpectralFlux, MKL, Cosine), the distance does not have to be computed between consecutive frames. By default (0) it is, otherwise this sets the distane between the comparison window in samples.
For certain metrics (HFC, SpectralFlux, MKL, Cosine), the distance does not have to be computed between consecutive frames. By default (0) it is, otherwise this sets the distane between the comparison window in samples.
ARGUMENT:: windowSize
The window size. As spectral differencing relies on spectral frames, we need to decide what precision we give it spectrally and temporally, in line with Gabor Uncertainty principles. http://www.subsurfwiki.org/wiki/Gabor_uncertainty
@ -71,10 +71,10 @@ ARGUMENT:: fftSize
The inner FFT/IFFT size. It should be at least 4 samples long, at least the size of the window, and a power of 2. Making it larger allows an oversampling of the spectral precision. The -1 default value will default to windowSize.
ARGUMENT:: action
A Function to be evaluated once the offline process has finished and indices instance variables have been updated on the client side. The function will be passed indices as an argument.
A Function to be evaluated once the offline process has finished and indices instance variables have been updated on the client side. The metric will be passed indices as an argument.
RETURNS::
Nothing, as the various destination buffers are declared in the function call.
Nothing, as the destination buffer is declared in the function call.
## 2 || YinFFT: Implements the frequency domain version of the YIN algorithm, as described in FOOTNOTE::P. M. Brossier, "Automatic Annotation of Musical Audio for Interactive Applications.” QMUL, London, UK, 2007. :: See also https://essentia.upf.edu/documentation/reference/streaming_PitchYinFFT.html
::
ARGUMENT:: minFreq
The minimum frequency that the algorithm will search for an estimated fundamental. This sets the lowest value that will be generated.
ARGUMENT:: maxFreq
The maximum frequency that the algorithm will search for an estimated fundamental. This sets the highest value that will be generated.
ARGUMENT:: unit
The unit of the estimated value. The default of 0 is in Hz. A value of 1 will convert to MIDI note values.
ARGUMENT:: windowSize
The window size. As sinusoidal estimation relies on spectral frames, we need to decide what precision we give it spectrally and temporally, in line with Gabor Uncertainty principles. http://www.subsurfwiki.org/wiki/Gabor_uncertainty
@ -105,11 +114,11 @@ b.play
// create a buffer as destinations
c = Buffer.new(s);
//run the process on them
//run the process on them, with limited bandwidth and units in MIDI notes
This class implements a non-real-time slicer using an algorithm assessing novelty in the signal to estimate the slicing points. A novelty curve is being derived from running a kernel across the diagonal of the similarity matrix, and looking for peak of changes. It implements the seminal results published in 'Automatic Audio Segmentation Using a Measure of Audio Novelty' by J Foote. It is part of the Fluid Decomposition Toolkit of the FluCoMa project. footnote::This was made possible thanks to the FluCoMa project ( http://www.flucoma.org/ ) funded by the European Research Council ( https://erc.europa.eu/ ) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No 725899).::
The process will return a buffer which contains indices (in sample) of estimated starting points of different slices.
CLASSMETHODS::
METHOD:: process
This is the method that calls for the slicing to be calculated on a given source buffer.
ARGUMENT:: server
The server on which the buffers to be processed are allocated.
ARGUMENT:: source
The index of the buffer to use as the source material to be sliced through novelty identification. The different channels of multichannel buffers will be summed.
ARGUMENT:: startFrame
Where in the srcBuf should the slicing process start, in sample.
ARGUMENT:: numFrames
How many frames should be processed.
ARGUMENT:: startChan
For multichannel srcBuf, which channel should be processed.
ARGUMENT:: numChans
For multichannel srcBuf, how many channel should be summed.
ARGUMENT:: indices
The index of the buffer where the indices (in sample) of the estimated starting points of slices will be written. The first and last points are always the boundary points of the analysis.
ARGUMENT:: feature
The feature on which novelty is computed.
table::
##0 || Spectrum || todo
##1 || MFCC || todo
##2 || Pitch || todo
##3 || Loudness || todo
::
ARGUMENT:: kernelSize
The granularity of the window in which the algorithm looks for change, in samples. A small number will be sensitive to short term changes, and a large number should look for long term changes.
ARGUMENT:: threshold
The normalised threshold, between 0 an 1, on the novelty curve to consider it a segmentation point.
ARGUMENT:: filterSize
The size of a smoothing filter that is applied on the novelty curve. A larger filter filter size allows for cleaner cuts on very sharp changes.
ARGUMENT:: windowSize
The window size. As novelty estimation relies on spectral frames, we need to decide what precision we give it spectrally and temporally, in line with Gabor Uncertainty principles. http://www.subsurfwiki.org/wiki/Gabor_uncertainty
ARGUMENT:: hopSize
The window hop size. As novelty estimation relies on spectral frames, we need to move the window forward. It can be any size but low overlap will create audible artefacts.
ARGUMENT:: fftSize
The inner FFT/IFFT size. It should be at least 4 samples long, at least the size of the window, and a power of 2. Making it larger allows an oversampling of the spectral precision.
ARGUMENT:: action
A Function to be evaluated once the offline process has finished and indices instance variables have been updated on the client side. The function will be passed indices as an argument.
RETURNS::
Nothing, as the various destination buffers are declared in the function call.
EXAMPLES::
code::
// load some buffers
(
b = Buffer.read(s,File.realpath(FluidBufRTNoveltySlice.class.filenameSymbol).dirname.withTrailingSlash ++ "../AudioFiles/Tremblay-AaS-AcousticStrums-M.wav");
//check the number of slices: it is the number of frames in the transBuf minus the boundary index.
c.query;
//loops over a splice with the MouseX
(
{
BufRd.ar(1, b,
Phasor.ar(0,1,
BufRd.kr(1, c,
MouseX.kr(0, BufFrames.kr(c) - 1), 0, 1),
BufRd.kr(1, c,
MouseX.kr(1, BufFrames.kr(c)), 0, 1),
BufRd.kr(1,c,
MouseX.kr(0, BufFrames.kr(c) - 1), 0, 1)), 0, 1);
}.play;
)
::
STRONG::Examples of the impact of the filterSize::
CODE::
// load some buffers
(
b = Buffer.read(s,File.realpath(FluidBufRTNoveltySlice.class.filenameSymbol).dirname.withTrailingSlash ++ "../AudioFiles/Tremblay-AaS-AcousticStrums-M.wav");
// change the filterSize in the code above to 4. Then to 8. Listen in between to the differences.
// What's happening? In the first instance (filterSize = 1), the novelty line is jittery and therefore overtriggers on the arpegiated guitar. We also can hear attacks at the end of the segment. Setting the threshold higher (like in the 'Basic Example' pane) misses some more subtle variations.
// So in the second settings (filterSize = 4), we smooth the novelty line a little, which allows us to catch small differences that are not jittery. It also corrects the ending cutting by the same trick: the averaging of the sharp pick is sliding up, crossing the threshold slightly earlier.
// If we smooth too much, like the third settings (filterSize = 8), we start to loose precision. Have fun with different values of theshold then will allow you to find the perfect segment for your signal.
The index of the buffer where the residual of the sinusoidal component will be reconstructed.
ARGUMENT:: bandwidth
The width in bins of the fragment of the fft window that is considered a normal deviation for a potential continuous sinusoidal track. It has an effect on CPU cost: the widest is more accurate but more computationally expensive.
The width in bins of the fragment of the fft window that is considered a normal deviation for a potential continuous sinusoidal track. It has an effect on CPU cost: the widest is more accurate but more computationally expensive. It is capped at (fftSize / 2) + 1.
ARGUMENT:: threshold
The normalised threshold, between 0 an 1, to consider a peak as a sinusoidal component from the normalized cross-correlation.
@ -64,7 +64,7 @@ ARGUMENT:: hopSize
The window hop size. As sinusoidal estimation relies on spectral frames, we need to move the window forward. It can be any size but low overlap will create audible artefacts.
ARGUMENT:: fftSize
The inner FFT/IFFT size. It should be at least 4 samples long, at least the size of the window, and a power of 2. Making it larger allows an oversampling of the spectral precision.
The inner FFT/IFFT size. It should be at least 4 samples long, at least the size of the window, and a power of 2. Making it larger allows an oversampling of the spectral precision. The -1 default value will default to the highest of windowSize and (bandwidth - 1) * 2.
ARGUMENT:: action
A Function to be evaluated once the offline process has finished and all Buffer's instance variables have been updated on the client side. The function will be passed [sines, residual] as an argument.
This class implements many spectral based onset detection functions, most of them taken from the literature. (http://www.dafx.ca/proceedings/papers/p_133.pdf) Some are already available in SuperCollider's LINK::Classes/Onsets:: object. It is part of the Fluid Decomposition Toolkit of the FluCoMa project.footnote::This was made possible thanks to the FluCoMa project ( http://www.flucoma.org/ ) funded by the European Research Council ( https://erc.europa.eu/ ) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No 725899).::
This class implements a real-time slicer using an algorithm assessing novelty in the signal to estimate the slicing points. A novelty curve is being derived from running a kernel across the diagonal of the similarity matrix, and looking for peak of changes. It implements the seminal results published in 'Automatic Audio Segmentation Using a Measure of Audio Novelty' by J Foote. It is part of the Fluid Decomposition Toolkit of the FluCoMa project.footnote::This was made possible thanks to the FluCoMa project ( http://www.flucoma.org/ ) funded by the European Research Council ( https://erc.europa.eu/ ) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No 725899).::
The process will return an audio steam with sample-long impulses at estimated starting points of the different slices.
@ -20,10 +20,10 @@ ARGUMENT:: in
ARGUMENT:: feature
The feature on which novelty is computed.
table::
##0 || Spectrum || todo
##1 || MFCC || todo
##2 || Pitch || todo
##3 || Loudness || todo
##0 || Spectrum || The magnitude of the full spectrum.
The granularity of the window in which the algorithm looks for change, in samples. A small number will be sensitive to short term changes, and a large number should look for long term changes.
@ -53,27 +53,28 @@ ARGUMENT:: maxFilterSize
This cannot be modulated.
RETURNS::
An audio stream with impulses at detected transients. The latency between the input and the output is windowSize at maximum.
An audio stream with impulses at detected transients. The latency between the input and the output is (windowSize +
This class implements many spectral based onset detection functions, most of them taken from the literature. (http://www.dafx.ca/proceedings/papers/p_133.pdf) Some are already available in SuperCollider's LINK::Classes/Onsets:: object. It is part of the Fluid Decomposition Toolkit of the FluCoMa project.footnote::This was made possible thanks to the FluCoMa project ( http://www.flucoma.org/ ) funded by the European Research Council ( https://erc.europa.eu/ ) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No 725899).::
This class implements many spectral based onset detection metrics, most of them taken from the literature. (http://www.dafx.ca/proceedings/papers/p_133.pdf) Some are already available in SuperCollider's LINK::Classes/Onsets:: object. It is part of the Fluid Decomposition Toolkit of the FluCoMa project.footnote::This was made possible thanks to the FluCoMa project ( http://www.flucoma.org/ ) funded by the European Research Council ( https://erc.europa.eu/ ) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No 725899).::
The process will return an audio steam with sample-long impulses at estimated starting points of the different slices.
@ -17,8 +17,8 @@ METHOD:: ar
ARGUMENT:: in
The audio to be processed.
ARGUMENT:: function
The function used to derive a difference curve between spectral frames. It can be any of the following:
ARGUMENT:: metric
The metric used to derive a difference curve between spectral frames. It can be any of the following:
TABLE::
##0 || Energy || thresholds on (sum of squares of magnitudes / nBins) (like Onsets \power)
##1 || HFC || thresholds on (sum of (squared magnitudes * binNum) / nBins)
@ -33,7 +33,7 @@ ARGUMENT:: function
::
ARGUMENT:: threshold
The thresholding of a new slice. Value ranges are different for each function, from 0 upwards.
The thresholding of a new slice. Value ranges are different for each metric, from 0 upwards.
ARGUMENT:: minSliceLength
The minimum duration of a slice in number of hopSize.
@ -42,7 +42,7 @@ ARGUMENT:: filterSize
The size of a smoothing filter that is applied on the novelty curve. A larger filter filter size allows for cleaner cuts on very sharp changes.
ARGUMENT:: frameDelta
For certain functions (HFC, SpectralFlux, MKL, Cosine), the distance does not have to be computed between consecutive frames. By default (0) it is, otherwise this sets the distane between the comparison window in samples.
For certain metrics (HFC, SpectralFlux, MKL, Cosine), the distance does not have to be computed between consecutive frames. By default (0) it is, otherwise this sets the distane between the comparison window in samples.
ARGUMENT:: windowSize
The window size. As sinusoidal estimation relies on spectral frames, we need to decide what precision we give it spectrally and temporally, in line with Gabor Uncertainty principles. http://www.subsurfwiki.org/wiki/Gabor_uncertainty
## 2 || YinFFT: Implements the frequency domain version of the YIN algorithm, as described in FOOTNOTE::P. M. Brossier, "Automatic Annotation of Musical Audio for Interactive Applications.” QMUL, London, UK, 2007. :: See also https://essentia.upf.edu/documentation/reference/streaming_PitchYinFFT.html
::
ARGUMENT:: minFreq
The minimum frequency that the algorithm will search for an estimated fundamental. This sets the lowest value that will be generated.
ARGUMENT:: maxFreq
The maximum frequency that the algorithm will search for an estimated fundamental. This sets the highest value that will be generated.
ARGUMENT:: unit
The unit of the estimated value. The default of 0 is in Hz. A value of 1 will convert to MIDI note values.
ARGUMENT:: windowSize
The window size. As sinusoidal estimation relies on spectral frames, we need to decide what precision we give it spectrally and temporally, in line with Gabor Uncertainty principles. http://www.subsurfwiki.org/wiki/Gabor_uncertainty
The width in bins of the fragment of the fft window that is considered a normal deviation for a potential continuous sinusoidal track. It has an effect on CPU cost: the widest is more accurate but more computationally expensive.
The width in bins of the fragment of the fft window that is considered a normal deviation for a potential continuous sinusoidal track. It has an effect on CPU cost: the widest is more accurate but more computationally expensive. It is capped at (fftSize / 2) + 1.
ARGUMENT:: threshold
The normalised threshold, between 0 an 1, to consider a peak as a sinusoidal component from the normalized cross-correlation.
@ -43,7 +43,7 @@ ARGUMENT:: hopSize
The window hop size. As sinusoidal estimation relies on spectral frames, we need to move the window forward. It can be any size but low overlap will create audible artefacts. The -1 default value will default to half of windowSize (overlap of 2).
ARGUMENT:: fftSize
The inner FFT/IFFT size. It should be at least 4 samples long, at least the size of the window, and a power of 2. Making it larger allows an oversampling of the spectral precision. The -1 default value will default to windowSize.
The inner FFT/IFFT size. It should be at least 4 samples long, at least the size of the window, and a power of 2. Making it larger allows an oversampling of the spectral precision. The -1 default value will default to the highest of windowSize and (bandwidth - 1) * 2.
ARGUMENT:: maxFFTSize
How large can the FFT be, by allocating memory at instantiation time. This cannot be modulated.
// A tool from the FluCoMa project, funded by the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No 725899)
// A tool from the FluCoMa project, funded by the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No 725899)
// A tool from the FluCoMa project, funded by the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No 725899)
// A tool from the FluCoMa project, funded by the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No 725899)
Owen Green
7 years ago