[텍스트마이닝]한국어 Word2vec+tsne

법률상담 데이터 분석(지식인 크롤링)

In [2]:

import pandas as pd

In [3]:

q = pd.read_csv('C:/q_title_utf.csv')

In [4]:

q.head()

Out[4]:

	title
0	양육비 미지급 청구와 파양 신고를 하고싶습니다.,
1	폭행 합의금을 못받고있습니다..,
2	재판과정 중 지속적인 문자로 공갈 협박을 받았습니다..,
3	토토운영책으로 불구속수사 중입니다 선고형량이 어느정도 수준일지 궁금합니다.,
4	인터넷 블로그에 후기를 작성했는데, 명예훼손 소송이 들어왔습니다,

In [5]:

title = list(q.title)
title[0:5]

Out[5]:

['양육비 미지급 청구와 파양 신고를 하고싶습니다.,',
 '폭행 합의금을 못받고있습니다..,',
 '재판과정 중 지속적인 문자로 공갈 협박을 받았습니다..,',
 '토토운영책으로 불구속수사 중입니다 선고형량이 어느정도 수준일지 궁금합니다.,',
 '인터넷 블로그에 후기를 작성했는데, 명예훼손 소송이 들어왔습니다,']

In [6]:

from string import punctuation
def strip_punctuation(s):
    return ''.join(c for c in s if c not in punctuation)

In [7]:

clean_title = []
for sent in title :
    clean = strip_punctuation(sent)
    clean_title.append(clean)

In [8]:

clean_title[0:3]

Out[8]:

['양육비 미지급 청구와 파양 신고를 하고싶습니다',
 '폭행 합의금을 못받고있습니다',
 '재판과정 중 지속적인 문자로 공갈 협박을 받았습니다']

w2v

In [9]:

from konlpy.tag import Twitter
twitter = Twitter()

In [10]:

twitter.nouns(clean_title[0])

Out[10]:

['양육비', '미지', '청구', '파양', '신고']

In [11]:

twitter.pos(clean_title[0])

Out[11]:

[('양육비', 'Noun'),
 ('미지', 'Noun'),
 ('급', 'Suffix'),
 ('청구', 'Noun'),
 ('와', 'Josa'),
 ('파양', 'Noun'),
 ('신고', 'Noun'),
 ('를', 'Josa'),
 ('하고', 'Verb'),
 ('싶', 'PreEomi'),
 ('습니다', 'Eomi')]

In [12]:

w2v_data = []

def tokenize(data):
    for sent in data:
        tokens = twitter.pos(sent)
        new_tokens = []
        for token in tokens:
            new_token = token[0]+'/'+token[1]
            new_tokens.append(new_token)
        w2v_data.append(new_tokens)
    return(w2v_data[0:2])

In [13]:

tokenize(clean_title)

Out[13]:

[['양육비/Noun',
  '미지/Noun',
  '급/Suffix',
  '청구/Noun',
  '와/Josa',
  '파양/Noun',
  '신고/Noun',
  '를/Josa',
  '하고/Verb',
  '싶/PreEomi',
  '습니다/Eomi'],
 ['폭행/Noun',
  '합의금/Noun',
  '을/Josa',
  '못/VerbPrefix',
  '받고/Verb',
  '있/PreEomi',
  '습니다/Eomi']]

In [53]:

import gensim
### from gensim.models import word2vec

num_features = 300    # Word vector dimensionality                      
min_word_count = 10   # Minimum word count                        
num_workers = 2     # Number of threads to run in parallel
context = 4          # Context window size                                                                                    
downsampling = 1e-3  # Downsample setting for frequent words

model = gensim.models.Word2Vec(w2v_data, workers=num_workers, 
                          size=num_features, min_count = min_word_count,
                          window = context, sample = downsampling)

model_name = "feature100_context2"
model.save(model_name)

In [57]:

model.most_similar("이혼/Noun", topn=10)

Out[57]:

[('협의/Noun', 0.9359733462333679),
 ('양육/Noun', 0.893966794013977),
 ('명시/Noun', 0.85338294506073),
 ('이혼소송/Noun', 0.8457387089729309),
 ('양육비/Noun', 0.8412048816680908),
 ('재산/Noun', 0.8383169174194336),
 ('친권/Noun', 0.836300253868103),
 ('분할/Noun', 0.8324744701385498),
 ('권/Suffix', 0.8160423040390015),
 ('포기/Noun', 0.7636870741844177)]

In [55]:

model.most_similar("상속/Noun", topn=10)

Out[55]:

[('포기/Noun', 0.9328514933586121),
 ('분할/Noun', 0.9154326915740967),
 ('시/Noun', 0.9125145673751831),
 ('권/Suffix', 0.9038272500038147),
 ('재산/Noun', 0.8899577856063843),
 ('양육/Noun', 0.8879168033599854),
 ('친권/Noun', 0.8761050701141357),
 ('명시/Noun', 0.8718239665031433),
 ('유산/Noun', 0.8492531180381775),
 ('한정승인/Noun', 0.8490205407142639)]

In [60]:

model.most_similar("폭행/Noun", topn=10)

Out[60]:

[('쌍방/Noun', 0.9463545680046082),
 ('집단/Noun', 0.9155074954032898),
 ('가해자/Noun', 0.9145557880401611),
 ('단순/Noun', 0.9122264385223389),
 ('사건/Noun', 0.9101886749267578),
 ('특수/Noun', 0.9053332805633545),
 ('절도/Noun', 0.9007413387298584),
 ('상해/Noun', 0.8957297801971436),
 ('피해자/Noun', 0.8565932512283325),
 ('성추행/Noun', 0.8388371467590332)]

In [61]:

model.most_similar("사기/Noun", topn=10)

Out[61]:

[('대포통장/Noun', 0.9140422344207764),
 ('보이스피싱/Noun', 0.8741967678070068),
 ('계정/Noun', 0.8513360619544983),
 ('성매매/Noun', 0.84779953956604),
 ('당한/Adjective', 0.8446159958839417),
 ('만남/Noun', 0.8286716938018799),
 ('게임/Noun', 0.8250682353973389),
 ('중고/Noun', 0.8246036767959595),
 ('경찰/Noun', 0.8221980929374695),
 ('금융/Noun', 0.8196936845779419)]

시각화

In [58]:

from sklearn.manifold import TSNE
import matplotlib.pyplot as plt

def tsne_plot(model):
    labels = []
    tokens = []

    for word in model.wv.vocab:
        tokens.append(model[word])
        labels.append(word)
    
    tsne_model = TSNE(perplexity=40, n_components=2, init='pca', n_iter=2500, random_state=23)
    new_values = tsne_model.fit_transform(tokens)

    x = []
    y = []
    for value in new_values:
        x.append(value[0])
        y.append(value[1])
        
    plt.figure(figsize=(16, 16)) 
    for i in range(len(x)):
        plt.scatter(x[i],y[i])
        plt.annotate(labels[i],
                     xy=(x[i], y[i]),
                     xytext=(5, 2),
                     textcoords='offset points',
                     ha='right',
                     va='bottom')
    plt.show()

In [59]:

tsne_plot(model)

폰트 추가함

• C:\Users\ksg/AppData\Local\Programs\Python\Python36\Lib\site-packages\matplotlib\mpl-data\fonts\ttf
• 폰트 : H2GTRM

In [50]:

font_name = "c:/Users/ksg/AppData/Local/Programs/Python/Python36/Lib/site-packages/matplotlib/mpl-data/fonts/ttf/H2GTRM.ttf"
font_name = font_manager.FontProperties(fname=font_name).get_name()
rc('font', family=font_name)