|
|
TITLE:: FluidHPSS
|
|
|
SUMMARY:: Harmonic-Percussive Source Separation Using Median Filtering
|
|
|
CATEGORIES:: Libraries>FluidDecomposition
|
|
|
RELATED:: Guides/FluCoMa, Guides/FluidDecomposition
|
|
|
|
|
|
A FluidHPSS object performs Harmonic-Percussive Source Separation (HPSS) on the an audio input. The class performs HPSS as described in its original form footnote::
|
|
|
Fitzgerald, Derry. 2010. ‘Harmonic/Percussive Separation Using Median Filtering’. In Proceedings DaFx 10. https://arrow.dit.ie/argcon/67.
|
|
|
:: as well as a variation on the extension propsoed by Driedger et al. footnote::
|
|
|
Driedger, Jonathan, Meinard Uller, and Sascha Disch. 2014. ‘Extending Harmonic-Percussive Separation of Audio Signals’. In Proc. ISMIR. http://www.terasoft.com.tw/conf/ismir2014/proceedings/T110_127_Paper.pdf.
|
|
|
::
|
|
|
|
|
|
The algorithm takes an audio in, and divides it into two or three outputs, depending on the mode: LIST::
|
|
|
## an harmonic component;
|
|
|
## a percussive component;
|
|
|
## a residual of the previous two if the flag is set to inter-dependant thresholds. See the maskingMode below.::
|
|
|
|
|
|
It is part of the Fluid Decomposition Toolkit of the FluCoMa project. footnote::
|
|
|
This was made possible thanks to the FluCoMa project ( http://www.flucoma.org/ ) funded by the European Research Council ( https://erc.europa.eu/ ) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No 725899).
|
|
|
::
|
|
|
|
|
|
More information on median filtering, and on HPSS for musicianly usage, are availabe in LINK::Guides/FluCoMa:: overview file.
|
|
|
CLASSMETHODS::
|
|
|
|
|
|
METHOD:: ar
|
|
|
The audio rate version of the object.
|
|
|
|
|
|
ARGUMENT:: in
|
|
|
The input to be processed.
|
|
|
|
|
|
ARGUMENT:: harmFilterSize
|
|
|
The size, in spectral frames, of the median filter for the harmonic component. Must be an odd number, >= 3.
|
|
|
|
|
|
ARGUMENT:: percFilterSize
|
|
|
The size, in spectral bins, of the median filter for the percussive component. Must be an odd number, >=3
|
|
|
|
|
|
ARGUMENT:: maskingMode
|
|
|
The way the masking is applied to the original spectrogram. (0,1,2)
|
|
|
table::
|
|
|
## 0 || The traditional soft mask used in Fitzgerald's original method of 'Wiener-inspired' filtering. Complimentary, soft masks are made for the harmonic and percussive parts by allocating some fraction of a point in time-frequency to each. This provides the fewest artefacts, but the weakest separation. The two resulting buffers will sum to exactly the original material.
|
|
|
## 1 || Relative mode - Better separation, with more artefacts. The harmonic mask is constructed using a binary decision, based on whether a threshold is exceeded at a given time-frequency point (these are set using harmThreshFreq1, harmThreshAmp1, harmThreshFreq2, harmThreshAmp2, see below). The percussive mask is then formed as the inverse of the harmonic one, meaning that as above, the two components will sum to the original sound.
|
|
|
|
|
|
## 2 || Inter-dependent mode - Thresholds can be varied independently, but are coupled in effect. Binary masks are made for each of the harmonic and percussive components, and the masks are converted to soft at the end so that everything null sums even if the params are independent, that is what makes it harder to control. These aren't guranteed to cover the whole sound; in this case the 'leftovers' will placed into a third buffer.
|
|
|
::
|
|
|
|
|
|
ARGUMENT:: harmThreshFreq1
|
|
|
In modes 1 and 2, the frequency of the low part of the threshold for the harmonic filter (0-1)
|
|
|
|
|
|
ARGUMENT:: harmThreshAmp1
|
|
|
In modes 1 and 2, the threshold of the low part for the harmonic filter. That threshold applies to all frequencies up to harmThreshFreq1: how much more powerful (in dB) the harmonic median filter needs to be than the percussive median filter for this bin to be counted as harmonic.
|
|
|
|
|
|
ARGUMENT:: harmThreshFreq2
|
|
|
In modes 1 and 2, the frequency of the hight part of the threshold for the harmonic filter. (0-1)
|
|
|
|
|
|
ARGUMENT:: harmThreshAmp2
|
|
|
In modes 1 and 2, the threshold of the high part for the harmonic filter. That threshold applies to all frequencies above harmThreshFreq2. The threshold between harmThreshFreq1 and harmThreshFreq2 is interpolated between harmThreshAmp1 and harmThreshAmp2. How much more powerful (in dB) the harmonic median filter needs to be than the percussive median filter for this bin to be counted as harmonic.
|
|
|
|
|
|
ARGUMENT:: percThreshFreq1
|
|
|
In mode 2, the frequency of the low part of the threshold for the percussive filter. (0-1)
|
|
|
|
|
|
ARGUMENT:: percThreshAmp1
|
|
|
In mode 2, the threshold of the low part for the percussive filter. That threshold applies to all frequencies up to percThreshFreq1. How much more powerful (in dB) the percussive median filter needs to be than the harmonic median filter for this bin to be counted as percussive.
|
|
|
|
|
|
ARGUMENT:: percThreshFreq2
|
|
|
In mode 2, the frequency of the hight part of the threshold for the percussive filter. (0-1)
|
|
|
|
|
|
ARGUMENT:: percThreshAmp2
|
|
|
In mode 2, the threshold of the high part for the percussive filter. That threshold applies to all frequencies above percThreshFreq2. The threshold between percThreshFreq1 and percThreshFreq2 is interpolated between percThreshAmp1 and percThreshAmp2. How much more powerful (in dB) the percussive median filter needs to be than the harmonic median filter for this bin to be counted as percussive.
|
|
|
|
|
|
ARGUMENT:: winSize
|
|
|
The window size. As sinusoidal estimation relies on spectral frames, we need to decide what precision we give it spectrally and temporally, in line with Gabor Uncertainty principles. http://www.subsurfwiki.org/wiki/Gabor_uncertainty
|
|
|
|
|
|
ARGUMENT:: hopSize
|
|
|
The window hope size. As sinusoidal estimation relies on spectral frames, we need to move the window forward. It can be any size but low overlap will create audible artefacts. The -1 default value will default to half of winSize (overlap of 2).
|
|
|
|
|
|
ARGUMENT:: fftSize
|
|
|
The inner FFT/IFFT size. It should be at least 4 samples long, at least the size of the window, and a power of 2. Making it larger allows an oversampling of the spectral precision. The -1 default value will default to windowSize.
|
|
|
|
|
|
ARGUMENT:: maxFFTSize
|
|
|
How large can the FFT be, by allocating memory at instantiation time. This cannot be modulated.
|
|
|
|
|
|
ARGUMENT::maxHarmFilterSize
|
|
|
How large can the harmonic filter be modulated to (harmFilterSize), by allocating memory at instantiation time. This cannot be modulated.
|
|
|
|
|
|
ARGUMENT:: maxPercFilterSize
|
|
|
How large can the percussive filter be modulated to (percFilterSize), by allocating memory at instantiation time. This cannot be modulated.
|
|
|
|
|
|
|
|
|
RETURNS::
|
|
|
An array of three audio streams: [0] is the harmonic part extracted, [1] is the percussive part extracted, [2] is the rest. The latency between the input and the output is ((harmFilterSize - 1) * hopSize) + winSize) samples.
|
|
|
|
|
|
|
|
|
Discussion::
|
|
|
HPSS works by using median filters on the spectral magnitudes of a sound. It hinges on a simple modelling assumption that tonal components will tend to yield concentrations of energy across time, spread out in frequency, and percussive components will manifest as concentrations of energy across frequency, spread out in time. By using median filters across time and frequency respectively, we get initial esitmates of the tonal-ness / transient-ness of a point in time and frequency. These are then combined into 'masks' that are applied to the orginal spectral data in order to produce a separation.
|
|
|
|
|
|
The maskingMode parameter provides different approaches to combinging estimates and producing masks. Some settings (especially in modes 1 & 2) will provide better separation but with more artefacts. These can, in principle, be ameliorated by applying smoothing filters to the masks before transforming back to the time-domain (not yet implemented).
|
|
|
|
|
|
|
|
|
EXAMPLES::
|
|
|
|
|
|
CODE::
|
|
|
//load a soundfile to play
|
|
|
b = Buffer.read(s,File.realpath(FluidHPSS.class.filenameSymbol).dirname.withTrailingSlash ++ "../AudioFiles/Tremblay-AaS-SynthTwoVoices-M.wav");
|
|
|
|
|
|
// run with basic parameters (left is harmonic, right is percussive)
|
|
|
{FluidHPSS.ar(PlayBuf.ar(1,b.bufnum,loop:1))}.play
|
|
|
|
|
|
// run in mode 1
|
|
|
{FluidHPSS.ar(PlayBuf.ar(1,b.bufnum,loop:1),17,51,1,0.05,40,0.1,-40)}.play
|
|
|
|
|
|
// run in mode 2, listening to:
|
|
|
//the harmonic stream
|
|
|
{FluidHPSS.ar(PlayBuf.ar(1,b.bufnum,loop:1),15,31,2,0.05,40,0.1,-40, 0.1, -10, 0.2, 10)[0].dup}.play
|
|
|
// the percussive stream
|
|
|
{FluidHPSS.ar(PlayBuf.ar(1,b.bufnum,loop:1),15,31,2,0.05,40,0.1,-40, 0.1, -10, 0.2, 10)[1].dup}.play
|
|
|
// the residual stream
|
|
|
{FluidHPSS.ar(PlayBuf.ar(1,b.bufnum,loop:1),15,31,2,0.05,40,0.1,-40, 0.1, -10, 0.2, 10)[2].dup}.play
|
|
|
|
|
|
// null test (the process add a latency of ((harmFilterSize - 1) * hopSize) + winSize) samples
|
|
|
{var sig = PlayBuf.ar(1,b.bufnum,loop:1); [FluidHPSS.ar(sig, 17, 31).sum - DelayN.ar(sig, 1, ((((17 - 1) * 512) + 1024) / s.sampleRate))]}.play
|
|
|
::
|