Unstructured data

Python में मशीन लर्निंग वर्कफ़्लो डिज़ाइन करना

Dr. Chris Anagnostopoulos

Honorary Associate Professor

Structured बनाम unstructured

  Class   AGE  SEX  STEROID    ...        
0    2.0  50.0  2.0      1.0    ...      
1    2.0  40.0  1.0      1.0    ...       
...

           label                                           sequence
0          VIRUS  AVTVVPDPTCCGTLSFKVPKDAKKGKHLGTFDIRQAIMDYGGLHSQ...
1  IMMUNE SYSTEM  QVQLQQPGAELVKPGASVKLSCKASGYTFTSYWMHWVKQRPGRGLE...
2  IMMUNE SYSTEM  QAVVTQESALTTSPGETVTLTCRSSTGAVTTSNYANWVQEKPDHLF...
3          VIRUS  MSQVTEQSVRFQTALASIKLIQASAVLDLTEDDFDFLTSNKVWIAT...
...

क्या हम ऐसा डिटेक्टर बना सकते हैं जो इस डेटा में वायरस को असामान्य के रूप में फ़्लैग करे?

import stringdist
stringdist.levenshtein('abc', 'acc')

stringdist.levenshtein('acc', 'cce')

             label   sequence
169  IMMUNE SYSTEM  ILSALVGIV
170  IMMUNE SYSTEM  ILSALVGIL

stringdist.levenshtein('ILSALVGIV', 'ILSALVGIL')

थोड़ा debugging

# This won't work
pdist(proteins['sequence'].iloc[:3], metric=stringdist.levenshtein)

Traceback (most recent call last):
ValueError: A 2-dimensional array must be passed.

थोड़ा debugging

sequences = np.array(proteins['sequence'].iloc[:3]).reshape(-1,1)

# This won't work for a different reason
pdist(sequences, metric=stringdist.levenshtein)

Traceback (most recent call last):
TypeError: argument 1 must be str, not numpy.ndarray

थोड़ा debugging

# This one works!!
def my_levenshtein(x, y):
    return stringdist.levenshtein(x[0], y[0])

pdist(sequences, metric=my_levenshtein)

array([136.,   2., 136.])

Precomputed मैट्रिक्स से प्रोटीन आउट्लायर

# This takes 2 minutes for about 1000 examples
M = pdist(sequences, my_levenshtein)

LoF डिटेक्टर, precomputed distance मैट्रिक्स के साथ:

# This takes 3 seconds
detector = lof(metric='precomputed', contamination=0.1)
preds = detector.fit_predict(M)

roc_auc_score(proteins['label'] == 'VIRUS', preds == -1)

0.64

अपनी distance चुनें

Python में मशीन लर्निंग वर्कफ़्लो डिज़ाइन करना