LDA topic modeling

Trains a latent Dirichlet allocation model with scikit-learn using abstracts from Neurosynth.

import os

import pandas as pd

from nimare import annotate
from nimare.nimads import Studyset
from nimare.utils import get_resource_path

Load Studyset with abstracts

studyset = Studyset(
    os.path.join(get_resource_path(), "neurosynth_laird_studyset.json"),
    target="mni152_2mm",
)

Initialize LDA model

model = annotate.lda.LDAModel(n_topics=5, max_iter=1000, text_column="abstract")

Run model

new_studyset = model.fit(studyset)

View results

This DataFrame is very large, so we will only show a slice of it.

new_studyset.annotations_df[new_studyset.annotations_df.columns[:10]].head(10)

	id	study_id	contrast_id	LDA5__1_connectivity_functional_cognitive	LDA5__2_connectivity_functional_posterior	LDA5__3_connectivity_functional_networks	LDA5__4_social_cortex_functional	LDA5__5_literature_identified_stimulation	Neurosynth_TFIDF__001	Neurosynth_TFIDF__01
0	17029760-1	17029760	1	0.004191	0.004180	0.004173	0.004194	0.983261	0.0	0.0
1	18760263-1	18760263	1	0.008047	0.008009	0.008087	0.967781	0.008077	0.0	0.0
2	19162389-1	19162389	1	0.007242	0.823798	0.007210	0.007257	0.154493	0.0	0.0
3	19603407-1	19603407	1	0.239845	0.003527	0.749681	0.003482	0.003465	0.0	0.0
4	20197097-1	20197097	1	0.005102	0.005054	0.005042	0.462767	0.522034	0.0	0.0
5	22569543-1	22569543	1	0.002697	0.989114	0.002723	0.002701	0.002766	0.0	0.0
6	22659444-1	22659444	1	0.001517	0.993941	0.001529	0.001511	0.001502	0.0	0.0
7	23042731-1	23042731	1	0.474882	0.404157	0.115939	0.002511	0.002511	0.0	0.0
8	23702412-1	23702412	1	0.002208	0.134883	0.272855	0.587866	0.002188	0.0	0.0
9	24681401-1	24681401	1	0.979231	0.005183	0.005187	0.005191	0.005209	0.0	0.0

Given that this DataFrame is very wide (many terms), we will transpose it before presenting it.

model.distributions_["p_topic_g_word_df"].T.head(10)

	LDA5__1_connectivity_functional_cognitive	LDA5__2_connectivity_functional_posterior	LDA5__3_connectivity_functional_networks	LDA5__4_social_cortex_functional	LDA5__5_literature_identified_stimulation
10	0.001000	0.001000	0.001000	2.000941	0.001000
abstract	1.001177	0.001000	0.001000	1.000731	0.001000
action	0.001000	2.000757	0.001000	0.001000	0.001000
active	0.001000	0.001000	1.000852	2.000337	1.001695
addition	1.000676	0.001000	1.000532	1.092715	1.909921
additionally	2.000963	0.001000	0.001000	0.001000	0.001000
affective	0.001000	1.892452	1.338409	2.771964	0.001000
affective processes	0.001000	2.000902	0.001000	0.001000	0.001000
ale	0.001000	0.001000	1.000751	0.001000	1.001164
altered	1.000686	0.001000	3.001244	0.001000	0.001000

n_top_terms = 10
top_term_df = model.distributions_["p_topic_g_word_df"].T
temp_df = top_term_df.copy()
top_term_df = pd.DataFrame(columns=top_term_df.columns, index=range(n_top_terms))
top_term_df.index.name = "Token"
for col in top_term_df.columns:
    top_tokens = temp_df.sort_values(by=col, ascending=False).index.tolist()[:n_top_terms]
    top_term_df.loc[:, col] = top_tokens

top_term_df

	LDA5__1_connectivity_functional_cognitive	LDA5__2_connectivity_functional_posterior	LDA5__3_connectivity_functional_networks	LDA5__4_social_cortex_functional	LDA5__5_literature_identified_stimulation
Token
0	connectivity	connectivity	connectivity	social	literature
1	functional	functional	functional	cortex	identified
2	cognitive	posterior	networks	functional	stimulation
3	behavioral	motor	functional networks	human	talairach
4	network	functional connectivity	anterior	frontal	published
5	functions	macm	insula	lateral	coordinate
6	functional connectivity	seed	macm	cognition	magnetic
7	modeling	task	function	network	prefrontal
8	error	anterior	method	frontal pole	resting
9	dorsal	human	structural	lobe	use