Friday, September 2, 2011

09/01/2011

When you make the shingle lower, than the precision will be decrease to get the similarity of documents because the key words do not need to be consecutive. In other words, you should use more shingles to compare the similarity of dicuments. But, you may also note that sometimes, it may lead two documents not relevant if you use some irrelevant shingles. Hence, make the trade off between them.

Because customs may under speicfy or over specify what they are looking for, so you cannot thow a document even if miss sth.

How do I know which word is more important? You need to consider the distribution of the word over the documents and figure it out how common it is. Also, the documents of the similartiy should not depend on the size of documents. The meaning will not change if you dobule the size.
-- Shu Wang