LDA topic modeling

Trains a latent Dirichlet allocation model with scikit-learn using abstracts from Neurosynth.

import os

import pandas as pd

from nimare import annotate
from nimare.nimads import Studyset
from nimare.utils import get_resource_path

Load Studyset with abstracts

studyset = Studyset(
    os.path.join(get_resource_path(), "neurosynth_laird_studyset.json"),
    target="mni152_2mm",
)

Initialize LDA model

model = annotate.lda.LDAModel(n_topics=5, max_iter=1000, text_column="abstract")

Run model

new_studyset = model.fit(studyset)

View results

This DataFrame is very large, so we will only show a slice of it.

new_studyset.annotations_df[new_studyset.annotations_df.columns[:10]].head(10)

	id	study_id	contrast_id	Neurosynth_TFIDF__001	Neurosynth_TFIDF__01	Neurosynth_TFIDF__05	Neurosynth_TFIDF__10	Neurosynth_TFIDF__100	Neurosynth_TFIDF__11	Neurosynth_TFIDF__12
0	17029760-1	17029760	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0
1	18760263-1	18760263	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0
2	19162389-1	19162389	1	0.0	0.0	0.0	0.000000	0.0	0.176321	0.0
3	19603407-1	19603407	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0
4	20197097-1	20197097	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0
5	22569543-1	22569543	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0
6	22659444-1	22659444	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0
7	23042731-1	23042731	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0
8	23702412-1	23702412	1	0.0	0.0	0.0	0.061006	0.0	0.000000	0.0
9	24681401-1	24681401	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0

Given that this DataFrame is very wide (many terms), we will transpose it before presenting it.

model.distributions_["p_topic_g_word_df"].T.head(10)

	LDA5__1_cortex_motor_identified	LDA5__2_connectivity_functional_human	LDA5__3_connectivity_functional_posterior	LDA5__4_social_functional_connectivity	LDA5__5_literature_error_talairach
10	0.001000	1.000808	0.001000	1.001060	0.001000
abstract	1.000975	0.001000	0.001000	1.000935	0.001000
action	0.001000	1.000830	1.001037	0.001000	0.001000
active	1.000830	1.000666	0.001000	2.001337	0.001000
addition	3.001393	1.000281	0.001000	0.001000	1.001189
additionally	0.001000	1.000441	0.001000	0.001000	1.001424
affective	0.001000	3.000786	1.000928	2.001143	0.001000
affective processes	0.001000	1.000701	1.001137	0.001000	0.001000
ale	0.001000	1.000477	0.001000	0.001000	1.001417
altered	0.001000	3.000751	0.001000	0.001000	1.001172

n_top_terms = 10
top_term_df = model.distributions_["p_topic_g_word_df"].T
temp_df = top_term_df.copy()
top_term_df = pd.DataFrame(columns=top_term_df.columns, index=range(n_top_terms))
top_term_df.index.name = "Token"
for col in top_term_df.columns:
    top_tokens = temp_df.sort_values(by=col, ascending=False).index.tolist()[:n_top_terms]
    top_term_df.loc[:, col] = top_tokens

top_term_df

	LDA5__1_cortex_motor_identified	LDA5__2_connectivity_functional_human	LDA5__3_connectivity_functional_posterior	LDA5__4_social_functional_connectivity	LDA5__5_literature_error_talairach
Token
0	cortex	connectivity	connectivity	social	literature
1	motor	functional	functional	functional	error
2	identified	human	posterior	connectivity	talairach
3	published	networks	functional connectivity	method	control
4	cognitive	approaches	macm	frontal	functional
5	connectivity	structural	anterior	human	using
6	functional	functional networks	task	using	network
7	stimulation	anterior	connections	analytic	suggest
8	dorsal	insula	seed	pole	coordinate
9	lateral	maps	methods	frontal pole	specifically