LDA topic modeling

Trains a latent Dirichlet allocation model with scikit-learn using abstracts from Neurosynth.

import os

import pandas as pd

from nimare import annotate
from nimare.dataset import Dataset
from nimare.utils import get_resource_path

Load dataset with abstracts

dset = Dataset(os.path.join(get_resource_path(), "neurosynth_laird_studies.json"))

Initialize LDA model

model = annotate.lda.LDAModel(n_topics=5, max_iter=1000, text_column="abstract")

Run model

new_dset = model.fit(dset)

View results

This DataFrame is very large, so we will only show a slice of it.

new_dset.annotations[new_dset.annotations.columns[:10]].head(10)

	id	study_id	contrast_id	Neurosynth_TFIDF__001	Neurosynth_TFIDF__01	Neurosynth_TFIDF__05	Neurosynth_TFIDF__10	Neurosynth_TFIDF__100	Neurosynth_TFIDF__11	Neurosynth_TFIDF__12
0	17029760-1	17029760	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0
1	18760263-1	18760263	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0
2	19162389-1	19162389	1	0.0	0.0	0.0	0.000000	0.0	0.176321	0.0
3	19603407-1	19603407	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0
4	20197097-1	20197097	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0
5	22569543-1	22569543	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0
6	22659444-1	22659444	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0
7	23042731-1	23042731	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0
8	23702412-1	23702412	1	0.0	0.0	0.0	0.061006	0.0	0.000000	0.0
9	24681401-1	24681401	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0

Given that this DataFrame is very wide (many terms), we will transpose it before presenting it.

model.distributions_["p_topic_g_word_df"].T.head(10)

	LDA5__1_connectivity_functional_macm	LDA5__2_social_functional_connectivity	LDA5__3_human_functional_cortex	LDA5__4_functional_connectivity_cortex	LDA5__5_connectivity_functional_anterior
10	0.001000	0.001000	2.001000	0.001000	0.001000
abstract	0.001000	1.001017	1.000983	0.001000	0.001000
action	1.000992	0.001000	1.001008	0.001000	0.001000
active	2.000479	1.001290	0.001000	0.001000	1.001231
addition	0.001000	1.000907	1.000807	2.001293	1.000993
additionally	1.001101	0.001000	0.001000	1.000899	0.001000
affective	1.014948	2.292322	1.673929	0.001000	1.022801
affective processes	1.016165	0.001000	0.001000	0.001000	0.985835
ale	0.001000	1.001145	0.001000	1.000855	0.001000
altered	0.001000	3.001913	0.001000	1.000087	0.001000

n_top_terms = 10
top_term_df = model.distributions_["p_topic_g_word_df"].T
temp_df = top_term_df.copy()
top_term_df = pd.DataFrame(columns=top_term_df.columns, index=range(n_top_terms))
top_term_df.index.name = "Token"
for col in top_term_df.columns:
    top_tokens = temp_df.sort_values(by=col, ascending=False).index.tolist()[:n_top_terms]
    top_term_df.loc[:, col] = top_tokens

top_term_df

	LDA5__1_connectivity_functional_macm	LDA5__2_social_functional_connectivity	LDA5__3_human_functional_cortex	LDA5__4_functional_connectivity_cortex	LDA5__5_connectivity_functional_anterior
Token
0	connectivity	social	human	functional	connectivity
1	functional	functional	functional	connectivity	functional
2	macm	connectivity	cortex	cortex	anterior
3	method	identified	frontal	motor	posterior
4	patterns	stimulation	cognitive	network	insula
5	connections	function	cbp	literature	seed
6	human	resting	functions	structural	functional connectivity
7	behavioral	altered	maps	correlations	networks
8	methods	using	connectivity	control	functional networks
9	connectivity patterns	structural	parcellation	error	cognitive