LDA topic modeling

Trains a latent Dirichlet allocation model with scikit-learn using abstracts from Neurosynth.

import os

import pandas as pd

from nimare import annotate
from nimare.dataset import Dataset
from nimare.utils import get_resource_path

Load dataset with abstracts

dset = Dataset(os.path.join(get_resource_path(), "neurosynth_laird_studies.json"))

Initialize LDA model

model = annotate.lda.LDAModel(n_topics=5, max_iter=1000, text_column="abstract")

Run model

new_dset = model.fit(dset)

View results

This DataFrame is very large, so we will only show a slice of it.

new_dset.annotations[new_dset.annotations.columns[:10]].head(10)

	id	study_id	contrast_id	Neurosynth_TFIDF__001	Neurosynth_TFIDF__01	Neurosynth_TFIDF__05	Neurosynth_TFIDF__10	Neurosynth_TFIDF__100	Neurosynth_TFIDF__11	Neurosynth_TFIDF__12
0	17029760-1	17029760	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0
1	18760263-1	18760263	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0
2	19162389-1	19162389	1	0.0	0.0	0.0	0.000000	0.0	0.176321	0.0
3	19603407-1	19603407	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0
4	20197097-1	20197097	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0
5	22569543-1	22569543	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0
6	22659444-1	22659444	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0
7	23042731-1	23042731	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0
8	23702412-1	23702412	1	0.0	0.0	0.0	0.061006	0.0	0.000000	0.0
9	24681401-1	24681401	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0

Given that this DataFrame is very wide (many terms), we will transpose it before presenting it.

model.distributions_["p_topic_g_word_df"].T.head(10)

	LDA5__1_connectivity_functional_posterior	LDA5__2_connectivity_macm_method	LDA5__3_functional_cortex_human	LDA5__4_connectivity_functional_social	LDA5__5_10_neuropsychiatric_new
10	0.001000	0.001000	2.001000	0.001000	0.001
abstract	1.001019	0.001000	0.001000	1.000981	0.001
action	0.001000	1.001109	1.000891	0.001000	0.001
active	0.001000	0.001000	4.001000	0.001000	0.001
addition	2.001017	0.001000	3.000983	0.001000	0.001
additionally	2.001000	0.001000	0.001000	0.001000	0.001
affective	1.344472	0.001000	2.657548	2.000980	0.001
affective processes	2.001000	0.001000	0.001000	0.001000	0.001
ale	0.001000	0.001000	1.000887	1.001113	0.001
altered	1.000319	0.001000	0.001000	3.001681	0.001

n_top_terms = 10
top_term_df = model.distributions_["p_topic_g_word_df"].T
temp_df = top_term_df.copy()
top_term_df = pd.DataFrame(columns=top_term_df.columns, index=range(n_top_terms))
top_term_df.index.name = "Token"
for col in top_term_df.columns:
    top_tokens = temp_df.sort_values(by=col, ascending=False).index.tolist()[:n_top_terms]
    top_term_df.loc[:, col] = top_tokens

top_term_df

	LDA5__1_connectivity_functional_posterior	LDA5__2_connectivity_macm_method	LDA5__3_functional_cortex_human	LDA5__4_connectivity_functional_social	LDA5__5_10_neuropsychiatric_new
Token
0	connectivity	connectivity	functional	connectivity	10
1	functional	macm	cortex	functional	pattern
2	posterior	method	human	social	neuropsychiatric
3	anterior	functional	frontal	structural	new
4	functional connectivity	patterns	memory	approaches	non
5	cognitive	behavioral	prefrontal	networks	nonhuman
6	cortex	connections	cognitive	insula	nonhuman primates
7	functions	methods	identified	correlations	number
8	task	connectivity patterns	maps	anterior	obtained
9	motor	nonhuman	literature	macm	order