Unstructured data

Designing Machine Learning Workflows in Python

Dr. Chris Anagnostopoulos

Honorary Associate Professor

Structured versus unstructured

  Class   AGE  SEX  STEROID    ...        
0    2.0  50.0  2.0      1.0    ...      
1    2.0  40.0  1.0      1.0    ...       
...

           label                                           sequence
0          VIRUS  AVTVVPDPTCCGTLSFKVPKDAKKGKHLGTFDIRQAIMDYGGLHSQ...
1  IMMUNE SYSTEM  QVQLQQPGAELVKPGASVKLSCKASGYTFTSYWMHWVKQRPGRGLE...
2  IMMUNE SYSTEM  QAVVTQESALTTSPGETVTLTCRSSTGAVTTSNYANWVQEKPDHLF...
3          VIRUS  MSQVTEQSVRFQTALASIKLIQASAVLDLTEDDFDFLTSNKVWIAT...
...

Can we build a detector that flags viruses as anomalous in this data?

import stringdist
stringdist.levenshtein('abc', 'acc')

stringdist.levenshtein('acc', 'cce')

             label   sequence
169  IMMUNE SYSTEM  ILSALVGIV
170  IMMUNE SYSTEM  ILSALVGIL

stringdist.levenshtein('ILSALVGIV', 'ILSALVGIL')

Some debugging

# This won't work
pdist(proteins['sequence'].iloc[:3], metric=stringdist.levenshtein)

Traceback (most recent call last):
ValueError: A 2-dimensional array must be passed.

Some debugging

sequences = np.array(proteins['sequence'].iloc[:3]).reshape(-1,1)

# This won't work for a different reason
pdist(sequences, metric=stringdist.levenshtein)

Traceback (most recent call last):
TypeError: argument 1 must be str, not numpy.ndarray

Some debugging

# This one works!!
def my_levenshtein(x, y):
    return stringdist.levenshtein(x[0], y[0])

pdist(sequences, metric=my_levenshtein)

array([136.,   2., 136.])

Protein outliers with precomputed matrices

# This takes 2 minutes for about 1000 examples
M = pdist(sequences, my_levenshtein)

LoF detector with a precomputed distance matrix:

# This takes 3 seconds
detector = lof(metric='precomputed', contamination=0.1)
preds = detector.fit_predict(M)

roc_auc_score(proteins['label'] == 'VIRUS', preds == -1)

0.64

Pick your distance

Designing Machine Learning Workflows in Python