LDA topic modeling

Trains a latent Dirichlet allocation model with scikit-learn using abstracts from Neurosynth.

import os

import pandas as pd

from nimare import annotate
from nimare.dataset import Dataset
from nimare.utils import get_resource_path

Load dataset with abstracts

dset = Dataset(os.path.join(get_resource_path(), "neurosynth_laird_studies.json"))

Initialize LDA model

model = annotate.lda.LDAModel(n_topics=5, max_iter=1000, text_column="abstract")

Run model

new_dset = model.fit(dset)

View results

This DataFrame is very large, so we will only show a slice of it.

new_dset.annotations[new_dset.annotations.columns[:10]].head(10)

	id	study_id	contrast_id	Neurosynth_TFIDF__001	Neurosynth_TFIDF__01	Neurosynth_TFIDF__05	Neurosynth_TFIDF__10	Neurosynth_TFIDF__100	Neurosynth_TFIDF__11	Neurosynth_TFIDF__12
0	17029760-1	17029760	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0
1	18760263-1	18760263	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0
2	19162389-1	19162389	1	0.0	0.0	0.0	0.000000	0.0	0.176321	0.0
3	19603407-1	19603407	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0
4	20197097-1	20197097	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0
5	22569543-1	22569543	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0
6	22659444-1	22659444	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0
7	23042731-1	23042731	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0
8	23702412-1	23702412	1	0.0	0.0	0.0	0.061006	0.0	0.000000	0.0
9	24681401-1	24681401	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0

Given that this DataFrame is very wide (many terms), we will transpose it before presenting it.

model.distributions_["p_topic_g_word_df"].T.head(10)

	LDA5__1_social_functional_correlations	LDA5__2_error_control_prefrontal cortex	LDA5__3_connectivity_functional_posterior	LDA5__4_connectivity_functional_anterior	LDA5__5_human_functional_maps
10	1.000942	0.001000	0.001000	0.001000	1.001058
abstract	1.000716	0.001000	0.001000	0.001000	1.001284
action	0.001000	0.001000	1.000899	0.001000	1.001101
active	0.999622	2.002936	0.001000	1.000442	0.001000
addition	2.001021	0.001000	0.001000	2.001114	1.000865
additionally	0.001000	1.001618	1.000382	0.001000	0.001000
affective	1.576802	0.001000	3.028973	0.001000	1.397224
affective processes	0.001000	0.001000	2.001000	0.001000	0.001000
ale	2.001000	0.001000	0.001000	0.001000	0.001000
altered	0.001000	1.001123	0.001000	0.001000	3.000877

n_top_terms = 10
top_term_df = model.distributions_["p_topic_g_word_df"].T
temp_df = top_term_df.copy()
top_term_df = pd.DataFrame(columns=top_term_df.columns, index=range(n_top_terms))
top_term_df.index.name = "Token"
for col in top_term_df.columns:
    top_tokens = temp_df.sort_values(by=col, ascending=False).index.tolist()[:n_top_terms]
    top_term_df.loc[:, col] = top_tokens

top_term_df

	LDA5__1_social_functional_correlations	LDA5__2_error_control_prefrontal cortex	LDA5__3_connectivity_functional_posterior	LDA5__4_connectivity_functional_anterior	LDA5__5_human_functional_maps
Token
0	social	error	connectivity	connectivity	human
1	functional	control	functional	functional	functional
2	correlations	prefrontal cortex	posterior	anterior	maps
3	identified	network	functional connectivity	insula	connectivity
4	literature	prefrontal	macm	networks	cognitive
5	motor	active	structural	approaches	functions
6	cortex	cortex	task	functional networks	cortex
7	published	parietal	patterns	lateral	function
8	stimulation	lobe	seed	cortex	dorsal
9	talairach	frontal	anterior	macm	structure