LDA topic modeling

Trains a latent Dirichlet allocation model with scikit-learn using abstracts from Neurosynth.

import os

import pandas as pd

from nimare import annotate
from nimare.nimads import Studyset
from nimare.utils import get_resource_path

Load Studyset with abstracts

studyset = Studyset(
    os.path.join(get_resource_path(), "neurosynth_laird_studyset.json"),
    target="mni152_2mm",
)

Initialize LDA model

model = annotate.lda.LDAModel(n_topics=5, max_iter=1000, text_column="abstract")

Run model

new_studyset = model.fit(studyset)

View results

This DataFrame is very large, so we will only show a slice of it.

new_studyset.annotations_df[new_studyset.annotations_df.columns[:10]].head(10)

	id	study_id	contrast_id	Neurosynth_TFIDF__001	Neurosynth_TFIDF__01	Neurosynth_TFIDF__05	Neurosynth_TFIDF__10	Neurosynth_TFIDF__100	Neurosynth_TFIDF__11	Neurosynth_TFIDF__12
0	17029760-1	17029760	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0
1	18760263-1	18760263	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0
2	19162389-1	19162389	1	0.0	0.0	0.0	0.000000	0.0	0.176321	0.0
3	19603407-1	19603407	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0
4	20197097-1	20197097	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0
5	22569543-1	22569543	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0
6	22659444-1	22659444	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0
7	23042731-1	23042731	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0
8	23702412-1	23702412	1	0.0	0.0	0.0	0.061006	0.0	0.000000	0.0
9	24681401-1	24681401	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0

Given that this DataFrame is very wide (many terms), we will transpose it before presenting it.

model.distributions_["p_topic_g_word_df"].T.head(10)

	LDA5__1_connectivity_functional_social	LDA5__2_human_seed_functional	LDA5__3_functional_connectivity_cortex	LDA5__4_connectivity_functional_functional connectivity	LDA5__5_working_working memory_10
10	0.001000	1.001271	1.000583	0.001000	0.001
abstract	1.000763	0.001000	0.001000	1.001140	0.001
action	1.000818	1.001096	0.001000	0.001000	0.001
active	2.001808	0.001000	2.000059	0.001000	0.001
addition	2.001403	0.001000	2.106287	0.895142	0.001
additionally	0.001000	0.001000	2.000911	0.001000	0.001
affective	1.000349	1.136290	3.866186	0.001000	0.001
affective processes	0.001000	0.001000	2.000896	0.001000	0.001
ale	0.001000	0.001000	2.000888	0.001000	0.001
altered	0.001000	0.001000	4.000893	0.001000	0.001

n_top_terms = 10
top_term_df = model.distributions_["p_topic_g_word_df"].T
temp_df = top_term_df.copy()
top_term_df = pd.DataFrame(columns=top_term_df.columns, index=range(n_top_terms))
top_term_df.index.name = "Token"
for col in top_term_df.columns:
    top_tokens = temp_df.sort_values(by=col, ascending=False).index.tolist()[:n_top_terms]
    top_term_df.loc[:, col] = top_tokens

top_term_df

	LDA5__1_connectivity_functional_social	LDA5__2_human_seed_functional	LDA5__3_functional_connectivity_cortex	LDA5__4_connectivity_functional_functional connectivity	LDA5__5_working_working memory_10
Token
0	connectivity	human	functional	connectivity	10
1	functional	seed	connectivity	functional	abstract
2	social	functional	cortex	functional connectivity	action
3	macm	maps	motor	structural	active
4	networks	cortex	posterior	correlations	addition
5	insula	voxel	anterior	functions	additionally
6	anterior	cognitive	task	characterized	affective
7	stimulation	anatomical	functional connectivity	resting state	affective processes
8	methods	correlates	using	state	ale
9	functional networks	obtained	human	resting	altered