LDA topic modeling

Trains a latent Dirichlet allocation model with scikit-learn using abstracts from Neurosynth.

import os

import pandas as pd

from nimare import annotate
from nimare.dataset import Dataset
from nimare.utils import get_resource_path

Load dataset with abstracts

dset = Dataset(os.path.join(get_resource_path(), "neurosynth_laird_studies.json"))

Initialize LDA model

model = annotate.lda.LDAModel(n_topics=5, max_iter=1000, text_column="abstract")

Run model

new_dset = model.fit(dset)

View results

This DataFrame is very large, so we will only show a slice of it.

new_dset.annotations[new_dset.annotations.columns[:10]].head(10)

	id	study_id	contrast_id	Neurosynth_TFIDF__001	Neurosynth_TFIDF__01	Neurosynth_TFIDF__05	Neurosynth_TFIDF__10	Neurosynth_TFIDF__100	Neurosynth_TFIDF__11	Neurosynth_TFIDF__12
0	17029760-1	17029760	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0
1	18760263-1	18760263	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0
2	19162389-1	19162389	1	0.0	0.0	0.0	0.000000	0.0	0.176321	0.0
3	19603407-1	19603407	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0
4	20197097-1	20197097	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0
5	22569543-1	22569543	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0
6	22659444-1	22659444	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0
7	23042731-1	23042731	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0
8	23702412-1	23702412	1	0.0	0.0	0.0	0.061006	0.0	0.000000	0.0
9	24681401-1	24681401	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0

Given that this DataFrame is very wide (many terms), we will transpose it before presenting it.

model.distributions_["p_topic_g_word_df"].T.head(10)

	LDA5__1_connectivity_functional_anterior	LDA5__2_motor_reflecting_cortex	LDA5__3_connectivity_functional_macm	LDA5__4_social_functional_connectivity	LDA5__5_functional_identified_literature
10	1.000859	0.001	0.001000	0.001000	1.001141
abstract	0.001000	0.001	1.000918	1.001082	0.001000
action	0.001000	0.001	2.001000	0.001000	0.001000
active	3.000914	0.001	0.001000	0.001000	1.001086
addition	1.201718	0.001	1.000575	0.001000	2.800706
additionally	1.000832	0.001	0.001000	1.001168	0.001000
affective	3.000743	0.001	0.001000	2.001489	1.000768
affective processes	2.001000	0.001	0.001000	0.001000	0.001000
ale	0.001000	0.001	0.001000	1.001132	1.000868
altered	0.001000	0.001	0.001000	4.001000	0.001000

n_top_terms = 10
top_term_df = model.distributions_["p_topic_g_word_df"].T
temp_df = top_term_df.copy()
top_term_df = pd.DataFrame(columns=top_term_df.columns, index=range(n_top_terms))
top_term_df.index.name = "Token"
for col in top_term_df.columns:
    top_tokens = temp_df.sort_values(by=col, ascending=False).index.tolist()[:n_top_terms]
    top_term_df.loc[:, col] = top_tokens

top_term_df

	LDA5__1_connectivity_functional_anterior	LDA5__2_motor_reflecting_cortex	LDA5__3_connectivity_functional_macm	LDA5__4_social_functional_connectivity	LDA5__5_functional_identified_literature
Token
0	connectivity	motor	connectivity	social	functional
1	functional	reflecting	functional	functional	identified
2	anterior	cortex	macm	connectivity	literature
3	human	presence	functional connectivity	network	published
4	networks	voxel	behavioral	structural	stimulation
5	posterior	compare	methods	altered	talairach
6	cortex	component	structural	control	cortex
7	insula	contribution	functions	function	frontal
8	functional connectivity	number	connections	error	parcellation
9	cognitive	obtained	approaches	modeling	using