LDA topic modeling

Trains a latent Dirichlet allocation model with scikit-learn using abstracts from Neurosynth.

import os

import pandas as pd

from nimare import annotate
from nimare.dataset import Dataset
from nimare.utils import get_resource_path

Load dataset with abstracts

dset = Dataset(os.path.join(get_resource_path(), "neurosynth_laird_studies.json"))

Initialize LDA model

model = annotate.lda.LDAModel(n_topics=5, max_iter=1000, text_column="abstract")

Run model

new_dset = model.fit(dset)

View results

This DataFrame is very large, so we will only show a slice of it.

new_dset.annotations[new_dset.annotations.columns[:10]].head(10)

	id	study_id	contrast_id	Neurosynth_TFIDF__001	Neurosynth_TFIDF__01	Neurosynth_TFIDF__05	Neurosynth_TFIDF__10	Neurosynth_TFIDF__100	Neurosynth_TFIDF__11	Neurosynth_TFIDF__12
0	17029760-1	17029760	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0
1	18760263-1	18760263	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0
2	19162389-1	19162389	1	0.0	0.0	0.0	0.000000	0.0	0.176321	0.0
3	19603407-1	19603407	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0
4	20197097-1	20197097	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0
5	22569543-1	22569543	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0
6	22659444-1	22659444	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0
7	23042731-1	23042731	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0
8	23702412-1	23702412	1	0.0	0.0	0.0	0.061006	0.0	0.000000	0.0
9	24681401-1	24681401	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0

Given that this DataFrame is very wide (many terms), we will transpose it before presenting it.

model.distributions_["p_topic_g_word_df"].T.head(10)

	LDA5__1_connectivity_anterior_functional	LDA5__2_connectivity_functional_macm	LDA5__3_social_human_functional	LDA5__4_cortex_motor_identified	LDA5__5_methods_connectivity patterns_mapping
10	0.001000	0.001000	2.001000	0.001000	0.001000
abstract	0.001000	1.000869	1.001131	0.001000	0.001000
action	0.001000	0.001000	1.000132	0.001000	1.001868
active	1.000961	0.001000	0.001000	3.001039	0.001000
addition	1.001072	1.000120	0.001000	3.001808	0.001000
additionally	0.001000	1.208691	0.001000	0.793309	0.001000
affective	0.001000	3.000533	3.001467	0.001000	0.001000
affective processes	0.001000	2.001000	0.001000	0.001000	0.001000
ale	0.001000	0.001000	0.001000	2.001000	0.001000
altered	0.001000	4.001000	0.001000	0.001000	0.001000

n_top_terms = 10
top_term_df = model.distributions_["p_topic_g_word_df"].T
temp_df = top_term_df.copy()
top_term_df = pd.DataFrame(columns=top_term_df.columns, index=range(n_top_terms))
top_term_df.index.name = "Token"
for col in top_term_df.columns:
    top_tokens = temp_df.sort_values(by=col, ascending=False).index.tolist()[:n_top_terms]
    top_term_df.loc[:, col] = top_tokens

top_term_df

	LDA5__1_connectivity_anterior_functional	LDA5__2_connectivity_functional_macm	LDA5__3_social_human_functional	LDA5__4_cortex_motor_identified	LDA5__5_methods_connectivity patterns_mapping
Token
0	connectivity	connectivity	social	cortex	methods
1	anterior	functional	human	motor	connectivity patterns
2	functional	macm	functional	identified	mapping
3	posterior	functional connectivity	frontal	prefrontal	cognition
4	insula	structural	maps	published	finally
5	networks	human	cognitive	literature	magnetic
6	functional networks	modeling	cortex	functional	brainmap
7	seed	patterns	frontal pole	stimulation	brainmap database
8	processes	behavioral	pole	prefrontal cortex	action
9	functional connectivity	connections	parcellation	talairach	particular