LDA topic modeling

Trains a latent Dirichlet allocation model with scikit-learn using abstracts from Neurosynth.

import os

import pandas as pd

from nimare import annotate
from nimare.dataset import Dataset
from nimare.utils import get_resource_path

Load dataset with abstracts

dset = Dataset(os.path.join(get_resource_path(), "neurosynth_laird_studies.json"))

Initialize LDA model

model = annotate.lda.LDAModel(n_topics=5, max_iter=1000, text_column="abstract")

Run model

new_dset = model.fit(dset)

View results

This DataFrame is very large, so we will only show a slice of it.

new_dset.annotations[new_dset.annotations.columns[:10]].head(10)

	id	study_id	contrast_id	Neurosynth_TFIDF__001	Neurosynth_TFIDF__01	Neurosynth_TFIDF__05	Neurosynth_TFIDF__10	Neurosynth_TFIDF__100	Neurosynth_TFIDF__11	Neurosynth_TFIDF__12
0	17029760-1	17029760	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0
1	18760263-1	18760263	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0
2	19162389-1	19162389	1	0.0	0.0	0.0	0.000000	0.0	0.176321	0.0
3	19603407-1	19603407	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0
4	20197097-1	20197097	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0
5	22569543-1	22569543	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0
6	22659444-1	22659444	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0
7	23042731-1	23042731	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0
8	23702412-1	23702412	1	0.0	0.0	0.0	0.061006	0.0	0.000000	0.0
9	24681401-1	24681401	1	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0

Given that this DataFrame is very wide (many terms), we will transpose it before presenting it.

model.distributions_["p_topic_g_word_df"].T.head(10)

	LDA5__1	LDA5__2	LDA5__3	LDA5__4	LDA5__5
10	0.001000	1.000809	1.001191	0.001000	0.001000
abstract	1.000777	0.001000	0.001000	0.001000	1.001223
action	0.001000	2.001000	0.001000	0.001000	0.001000
active	0.001000	1.089435	0.001000	2.912565	0.001000
addition	1.000467	1.089338	0.001000	2.913195	0.001000
additionally	0.001000	1.000448	0.001000	0.001000	1.001552
affective	1.348923	1.652207	2.001619	0.001000	1.001250
affective processes	2.001000	0.001000	0.001000	0.001000	0.001000
ale	0.001000	0.001000	1.001054	1.000946	0.001000
altered	0.001000	0.001000	3.001235	0.001000	1.000765

n_top_terms = 10
top_term_df = model.distributions_["p_topic_g_word_df"].T
temp_df = top_term_df.copy()
top_term_df = pd.DataFrame(columns=top_term_df.columns, index=range(n_top_terms))
top_term_df.index.name = "Token"
for col in top_term_df.columns:
    top_tokens = temp_df.sort_values(by=col, ascending=False).index.tolist()[:n_top_terms]
    top_term_df.loc[:, col] = top_tokens

top_term_df

	LDA5__1	LDA5__2	LDA5__3	LDA5__4	LDA5__5
Token
0	connectivity	connectivity	functional	cortex	social
1	functional	functional	connectivity	identified	network
2	motor	human	frontal	prefrontal	functional
3	posterior	macm	analytic	literature	error
4	functional connectivity	networks	structural	published	control
5	cognitive	approaches	function	talairach	task
6	anterior	functional networks	functionally	prefrontal cortex	modeling
7	task	anterior	parcellation	lateral	tasks
8	functions	cognitive	maps	cognition	systems
9	cortex	insula	cbp	parietal	connectivity