LDA topic modeling

Trains a latent Dirichlet allocation model with scikit-learn using abstracts from Neurosynth.

import os

import pandas as pd

from nimare import annotate
from nimare.nimads import Studyset
from nimare.utils import get_resource_path

Load Studyset with abstracts

studyset = Studyset(
    os.path.join(get_resource_path(), "neurosynth_laird_studyset.json"),
    target="mni152_2mm",
)

Initialize LDA model

model = annotate.lda.LDAModel(n_topics=5, max_iter=1000, text_column="abstract")

Run model

new_studyset = model.fit(studyset)

View results

This DataFrame is very large, so we will only show a slice of it.

new_studyset.annotations_df[new_studyset.annotations_df.columns[:10]].head(10)

	id	study_id	contrast_id	Neurosynth_TFIDF__001	Neurosynth_TFIDF__01	Neurosynth_TFIDF__05	Neurosynth_TFIDF__10	Neurosynth_TFIDF__100	Neurosynth_TFIDF__11	Neurosynth_TFIDF__12
0	17029760-1	17029760	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0
1	18760263-1	18760263	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0
2	19162389-1	19162389	1	0.0	0.0	0.0	0.000000	0.0	0.176321	0.0
3	19603407-1	19603407	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0
4	20197097-1	20197097	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0
5	22569543-1	22569543	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0
6	22659444-1	22659444	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0
7	23042731-1	23042731	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0
8	23702412-1	23702412	1	0.0	0.0	0.0	0.061006	0.0	0.000000	0.0
9	24681401-1	24681401	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0

Given that this DataFrame is very wide (many terms), we will transpose it before presenting it.

model.distributions_["p_topic_g_word_df"].T.head(10)

	LDA5__1_networks_insula_anterior	LDA5__2_connectivity_functional_functional connectivity	LDA5__3_motor_literature_functional	LDA5__4_social_cortex_frontal	LDA5__5_connectivity_structural_function
10	0.001000	0.001000	0.001000	2.000940	0.001000
abstract	0.001000	2.000890	0.001000	0.001000	0.001000
action	0.001000	2.000885	0.001000	0.001000	0.001000
active	2.001328	0.001000	0.001000	2.000611	0.001000
addition	1.959418	1.041702	2.001694	0.001000	0.001000
additionally	0.001000	1.000534	1.001275	0.001000	0.001000
affective	0.001000	3.069064	0.001000	1.932380	1.001417
affective processes	0.001000	2.000893	0.001000	0.001000	0.001000
ale	0.001000	0.001000	1.000737	0.001000	1.001216
altered	0.001000	0.001000	0.999631	0.001000	3.002328

n_top_terms = 10
top_term_df = model.distributions_["p_topic_g_word_df"].T
temp_df = top_term_df.copy()
top_term_df = pd.DataFrame(columns=top_term_df.columns, index=range(n_top_terms))
top_term_df.index.name = "Token"
for col in top_term_df.columns:
    top_tokens = temp_df.sort_values(by=col, ascending=False).index.tolist()[:n_top_terms]
    top_term_df.loc[:, col] = top_tokens

top_term_df

	LDA5__1_networks_insula_anterior	LDA5__2_connectivity_functional_functional connectivity	LDA5__3_motor_literature_functional	LDA5__4_social_cortex_frontal	LDA5__5_connectivity_structural_function
Token
0	networks	connectivity	motor	social	connectivity
1	insula	functional	literature	cortex	structural
2	anterior	functional connectivity	functional	frontal	function
3	functional networks	human	cortex	cognition	functional
4	functional	macm	network	functional	altered
5	connectivity	cognitive	error	lateral	indicate
6	identified	seed	control	human	functionally
7	stimulation	posterior	task	prefrontal cortex	structure function
8	memory	modeling	talairach	systems	structure
9	processes	task	indicated	pole	maps