TITLE:: FluidNMFFilter
SUMMARY:: Real-Time Non-Negative Matrix Factorisation with Fixed Bases
CATEGORIES:: Libraries>FluidDecomposition
RELATED:: Guides/FluCoMa, Guides/FluidDecomposition, Classes/FluidBufNMF, Classes/FluidNMFMatch

DESCRIPTION::
The FluidNMFFilter object decomposes and resynthesises an incoming audio signal against a set of spectral templates using an slimmed-down version of Nonnegative Matrix Factorisation (NMF) footnote:: Lee, Daniel D., and H. Sebastian Seung. 1999. ‘Learning the Parts of Objects by Non-Negative Matrix Factorization’. Nature 401 (6755): 788–91. https://doi.org/10.1038/44565. ::

It outputs at AR the resynthesis of the best factorisation. The spectral templates are presumed to have been produced by the offline NMF process (link::Classes/FluidBufNMF::), and must be the correct size with respect to the FFT settings being used (FFT size / 2 + 1 frames long). The rank of the decomposition (number of components) is determined by the number of channels in the supplied buffer of templates, up to a maximum set by the STRONG::maxComponents:: parameter.

NMF has been a popular technique in signal processing research for things like source separation and transcription footnote:: Smaragdis and Brown, Non-Negative Matrix Factorization for Polyphonic Music Transcription.::, although its creative potential is so far relatively unexplored. It works iteratively, by trying to find a combination of amplitudes ('activations') that yield the original magnitude spectrogram of the audio input when added together. By and large, there is no unique answer to this question (i.e. there are different ways of accounting for an evolving spectrum in terms of some set of templates and envelopes). In its basic form, NMF is a form of unsupervised learning: it starts with some random data and then converges towards something that minimizes the distance between its generated data and the original:it tends to converge very quickly at first and then level out. Fewer iterations mean less processing, but also less predictable results.

The whole process can be related to a channel vocoder where, instead of fixed bandpass filters, we get more complex filter shapes and the activations correspond to channel envelopes.

FluidBufNMF is part of the LINK:: Guides/FluidDecomposition:: of LINK:: Guides/FluCoMa::. For more explanations, learning material, and discussions on its musicianly uses, visit http://www.flucoma.org/


CLASSMETHODS::

METHOD:: ar
The real-time processing method. It takes an audio input, and will yield a audio stream in the form of a multichannel array of size STRONG::maxComponents:: . If the bases buffer has fewer than maxComponents channels, the remaining outputs will be zeroed.

ARGUMENT:: in
The signal input to the factorisation process.

ARGUMENT:: bases
	The server index of the buffer containing the different bases that the input signal will be matched against. Bases must be STRONG::(fft size / 2) + 1:: frames. If the buffer has more than STRONG::maxComponents:: channels, the excess will be ignored.

ARGUMENT::maxComponents
	The maximum number of elements the NMF algorithm will try to divide the spectrogram of the source in. This dictates the number of output channels for the ugen. This cannot be modulated.

ARGUMENT:: iterations
	The NMF process is iterative, trying to converge to the smallest error in its factorisation. The number of iterations will decide how many times it tries to adjust its estimates. Higher numbers here will be more CPU intensive, lower numbers will be more unpredictable in quality.

ARGUMENT:: windowSize
	The number of samples that are analysed at a time. A lower number yields greater temporal resolution, at the expense of spectral resoultion, and vice-versa.

ARGUMENT:: hopSize
	The window hop size. As NMF relies on spectral frames, we need to move the window forward. It can be any size but low overlap will create audible artefacts. The -1 default value will default to half of windowSize (overlap of 2).

ARGUMENT:: fftSize
	The FFT/IFFT size. It should be at least 4 samples long, at least the size of the window, and a power of 2. Making it larger allows an oversampling of the spectral precision. The -1 default value will use the next power of 2 equal or above the windowSize.

ARGUMENT:: maxFFTSize
	How large can the FFT be, by allocating memory at instantiation time. This cannot be modulated.

RETURNS::
	A multichannel kr output, giving for each basis component the activation amount.


EXAMPLES::
STRONG::A didactic example::
CODE::
(
// create buffers
b= Buffer.alloc(s,44100);
c = Buffer.alloc(s, 44100);
d = Buffer.new(s);
e= Buffer.new(s);
)

(
// fill them with 2 clearly segregated sine waves and composite a buffer where they are consecutive
Routine {
	b.sine2([500],[1], false, false);
	c.sine2([5000],[1],false, false);
	s.sync;
	FluidBufCompose.process(s,b, destination:d);
	FluidBufCompose.process(s,c, destStartFrame:44100, destination:d, destGain:1);
	s.sync;
	d.query;
}.play;
)

// check
d.plot
d.play //////(beware !!!! loud!!!)

(
// separate them in 2 components
Routine {
	FluidBufNMF.process(s, d, bases: e, components:2);
	s.sync;
	e.query;
}.play
)

// check for 2 spikes in the spectra
e.plot

//listen how the filter isolates each component and places them in each channel separately.
{FluidNMFFilter.ar(SinOsc.ar(500, mul: 0.1),e,2)}.play

{FluidNMFFilter.ar(SinOsc.ar(5000, mul: 0.1),e,2)}.play

{FluidNMFFilter.ar(SinOsc.ar([500,5000], mul: 0.1).sum,e,2)}.play
::

STRONG::A guitar processor::
CODE::
//set some buffers
(
b = Buffer.read(s,File.realpath(FluidNMFMatch.class.filenameSymbol).dirname.withTrailingSlash ++ "../AudioFiles/Tremblay-AaS-AcousticStrums-M.wav");
c = Buffer.new(s);
~bases = Buffer.new(s);
~spectralshapes = Buffer.new(s);
~stats = Buffer.new(s);
~centroids = Buffer.new(s);
~trainedBases = Buffer.new(s);
)

// train only 2 seconds
(
Routine {
    FluidBufNMF.process(s,b,0,88200,0,1, c, ~bases, components:10,fftSize:2048);
    c.query;
}.play;
)

// wait for the query to print
// find the component that has the picking sound checking the median spectral centroid
(
FluidBufSpectralShape.process(s, c, features: ~spectralshapes, action:{
	|shapes|FluidBufStats.process(s,shapes,stats:~stats, action:{
		|stats|stats.getn(0, (stats.numChannels * stats.numFrames) ,{
			|x| ~centroids = x.select({
				|item, index| (index.mod(7) == 0) && (index.div(70) == 5);
			})
		})
	})
});
)

// we then copy the basis with the lowest median centroid to a channel, and all the other bases to the other channel, of a 2-channel bases for decomposition
(
z = (0..9);
[z.removeAt(~centroids.minIndex)].do{|chan|FluidBufCompose.process(s, ~bases, startChan: chan, numChans: 1, destination: ~trainedBases, destGain:1)};
z.postln;
z.do({|chan| FluidBufCompose.process(s, ~bases, startChan:chan, numChans: 1, destStartChan: 1, destination: ~trainedBases, destGain:1)});
)

~trainedBases.plot;

//we can then use the resynthesised signal to sent in a delay
(
{
	var source, todelay, delay1, delay2, delay3, feedback, mod1, mod2, mod3, mod4;
	//read the source
	source = PlayBuf.ar(1, b);

	// generate modulators that are coprime in frequency
	mod1 = SinOsc.ar(1, 0, 0.001);
	mod2 = SinOsc.ar(((617 * 181) / (461 * 991)), 0, 0.001);
	mod3 = SinOsc.ar(((607 * 193) / (491 * 701)), 0, 0.001);
	mod4 = SinOsc.ar(((613 * 191) / (463 * 601)), 0, 0.001);

	// compress the signal to send to the delays
	todelay = FluidNMFFilter.ar(source,~trainedBases,2,fftSize:2048)[0]; //reading the channel of the activations on the pick basis

	// delay network
	feedback = LocalIn.ar(3);// take the feedback in for the delays
	delay1 = DelayC.ar(BPF.ar(todelay+feedback[1]+(feedback[2] * 0.3), 987, 6.7,0.8),0.123,0.122+(mod1*mod2));
	delay2 = DelayC.ar(BPF.ar(todelay+feedback[0]+(feedback[2] * 0.3), 1987, 6.7,0.8),0.345,0.344+(mod3*mod4));
	delay3 = DelayC.ar(BPF.ar(todelay+feedback[1], 1456, 6.7,0.8),0.567,0.566+(mod1*mod3),0.6);
	LocalOut.ar([delay1,delay2, delay3]); // write the feedback for the delays

			source.dup + ([delay1+delay3,delay2+delay3]*(3.dbamp))
	//listen to the delays in solo by uncommenting the following line
	// [delay1+delay3,delay2+delay3]
			// [source, todelay]
}.play;
)

::
STRONG::Strange Processor::
	CODE::
//set some buffers
(
b = Buffer.read(s,File.realpath(FluidNMFMatch.class.filenameSymbol).dirname.withTrailingSlash ++ "../AudioFiles/Nicol-LoopE-M.wav");
c = Buffer.alloc(s,1025,3);
d = Buffer.alloc(s,44100);
)

// play a source and circular record the last second, continuously
(
e = { var source = PlayBuf.ar(1,b,loop:1);
	BufWr.ar(source, d, Phasor.ar(1, end:44100));
	source.dup;
}.play;
)

// after at least 1 second, trigger a first factorisation
(
Routine {
    FluidBufNMF.process(s, d, bases:c, windowSize:2048, components:3);
    c.query;
}.play;
)
c.plot

// wait for the query to print
// then start the splitting effect
(
f = {var source = In.ar(0,2);
	ReplaceOut.ar(0, Splay.ar(FluidNMFFilter.ar(source.sum, c, 3, windowSize:2048)));
}.play(addAction:\addToTail);
)

// kill this boring splitter
f.free;

// more fun: processing the 3 component independently
(
f = {arg bases = c.bufnum;
			var source, x,y,z, rev, dist;
	source = In.ar(0,2);
			#x,y,z = FluidNMFFilter.ar(source.sum, bases, 3, windowSize:2048);
			rev = FreeVerb.ar(x);
			dist = (z * 10).atan * 0.1;
			ReplaceOut.ar(0, Splay.ar([rev,y,dist]));
}.play(addAction:\addToTail);
)

// set the bases
f.set(\bases, c.bufnum)

// here you can retrigger the factorisation
g = Buffer.alloc(s,1025,3);
FluidBufNMF.process(s, d, bases:g, windowSize:2048, components:3);
f.set(\bases, g.bufnum)

//free
f.free; e.free; b.free; c.free; d.free; g.free;
::
