[mlpack] K-means with HMM

[Ryan Curtin]

On Fri, May 16, 2014 at 10:56:03AM -0300, Thuener Silva wrote:
> Hello I'm using mlpack for research project in PUC-Rio. I'm trying to use
> K-Means to get the initial parameters for HMM to do an unsupervised
> learning. I noticed the phrase "I won't use K-Means because we can't afford
> to add the instability of that to our test." in one of the test cases of
> the hmm_test.cpp so you guys probably already done something like that.
> Can some one send me the some code example with that(K-Mean with HMM) or
> tell me where can I find that? It will help a lot.

Hi Thuener,

The reason we avoided using k-means in the test was that k-means can
sometimes produce very bad results (depending on the starting
positions), so taking that factor out of the test helps test HMMs
better.

I am assuming that what you want to do is use k-means to initialize
the means of the emission distributions of your HMM; and I'll assume
that like in the test you referenced, the emission distribution type is
the Gaussian distribution.  If that's not the case, please write back
and clarify so I can give better advice.

----
// This is our training dataset; we don't have any labels, so we are
// doing unsupervised training of the HMM.
extern std::vector<arma::mat> observations;

// Create the HMM object.
HMM<GaussianDistribution> hmm(numStates, GaussianDistribution(dimensionality));

// Reshape the vector of observations into one big matrix.
arma::mat data;
size_t totalCols = 0;
for (size_t i = 0; i < observations.size(); ++i)
  totalCols += observations[i].n_cols;

data.set_size(dimensionality, totalCols);
size_t startCol = 0;
for (size_t i = 0; i < observations.size(); ++i)
  data.submat(0, startCol, dimensionality - 1, startCol + observations[i].n_cols - 1) = observations[i];

// Now run k-means on the observations to get a set of initial means for
// the emission distributions.
arma::mat centroids;
arma::Col<size_t> observations; // We won't use this.
KMeans<> k(); // All default options.
k.Cluster(data, numStates, observations, centroids);

// Set the emission distribution's mean to the centroid of each cluster.
for (size_t i = 0; i < numStates; ++i)
  hmm.Emission()[i].Mean() = centroids.col(i);

// Ok, finally we are ready for training.
hmm.Train(observations);
----

I haven't compiled this and tested it, but I think it should work.  At
the very least, it should point you in the right direction about how to
use k-means to initialize your emission distributions.

If you're trying to use GMMs as your emission type instead of Gaussian
distributions, this process becomes a little more difficult.
Personally, I think that training GMM HMMs on unlabeled data is
unrealistic because there are so many free parameters.

If I can clarify anything I've written, please let me know.

Thanks!

Ryan

-- 
Ryan Curtin    | "Happy premise #2: There is no giant foot trying
ryan at ratml.org | to squash me." - Kit Ramsey