분류를 위한 특성 데이터 생성

Python에서 Spark SQL 입문

Mark Plutowski

Data Scientist

밀집 배열 변환하기

from pyspark.sql.functions import udf
from pyspark.sql.types import IntegerType
bad_udf = udf(lambda x:
              x.indices[0] 
              if (x and hasattr(x, "toArray") and x.numNonzeros()) 
              else 0,
              IntegerType())

밀집 배열 변환하기

try:
    df.select(bad_udf('outvec').alias('label')).first()
except Exception as e:
    print(e.__class__)
    print(e.errmsg)

<class 'py4j.protocol.Py4JJavaError'>
An error occurred while calling o90.collectToPython.

UDF 반환형은 올바르게 캐스팅해야 함

first_udf = udf(lambda x:
               int(x.indices[0]) 
               if (x and hasattr(x, "toArray") and x.numNonzeros())
               else 0,
               IntegerType())

동작 중인 UDF

+-------+--------------------+-----+--------------------+-------------------+
|endword|                 doc|count|            features|             outvec|
+-------+--------------------+-----+--------------------+-------------------+
|     it|[please, do, not,...| 1149|(12847,[15,47,502...|  (12847,[7],[1.0])|
| holmes|[start, of, the, ...|  107|(12847,[0,3,183,1...|(12847,[145],[1.0])|
|      i|[the, adventures,...|  103|(12847,[0,3,35,14...| (12847,[11],[1.0])|
+-------+--------------------+-----+--------------------+-------------------+

df.withColumn('label', k_udf('outvec')).drop('outvec').show(3)

+-------+--------------------+-----+--------------------+-----+
|endword|                 doc|count|            features|label|
+-------+--------------------+-----+--------------------+-----+
|     it|[please, do, not,...| 1149|(12847,[15,47,502...|    7|
| holmes|[start, of, the, ...|  107|(12847,[0,3,183,1...|  145|
|      i|[the, adventures,...|  103|(12847,[0,3,35,14...|   11|
+-------+--------------------+-----+--------------------+-----+

CountVectorizer

ETS: Extract Transform Select
CountVectorizer는 특성 추출기입니다.
입력은 문자열 배열입니다.
출력은 벡터입니다.

CountVectorizer 적합시키기

from pyspark.ml.feature import CountVectorizer

cv = CountVectorizer(inputCol='words',            
                     outputCol="features")
model = cv.fit(df)
result = model.transform(df)
print(result)

DataFrame[words: array<string>, features: vector]

# Dense string array on left, dense integer vector on right
+-------------------------+--------------------------------------+
|words                    |features                              |
+-------------------------+--------------------------------------+
|[Hello, world]           |(10,[7,9],[1.0,1.0])                  |
|[How, are, you?]         |(10,[1,3,4],[1.0,1.0,1.0])            |
|[I, am, fine, thank, you]|(10,[0,2,5,6,8],[1.0,1.0,1.0,1.0,1.0])|
+-------------------------+--------------------------------------+

연습해 봅시다!

Python에서 Spark SQL 입문