{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "9f89fade-5dbd-4a23-b8a1-f56f59667c04",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "## Creating a knowledge base for a custom GPT\n",
    "\n",
    "We created a custom GPT (Generative Pretrained Transformer) model, designed specifically to assist in the analysis and synthesis of information pertaining to the 31 reproducible research insights. The knowledge base of this retrieval-augmented generation framework comprises GPT-4 summaries of RRInsights interviews coupled with the original abstract, title, and keywords of the respective articles. This specialized GPT, named RRInsights, is tailored to process and interpret the provided data in the context of reproducibility. Through its advanced natural language processing capabilities, RRInsights can efficiently analyze the scoped literature, extract key insights, and generate comprehensive overviews of the research papers focusing on MRI technology."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e4fda273-04ba-4669-982c-b67d3063f2bf",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": [
     "hide-input"
    ]
   },
   "outputs": [],
   "source": [
    "import os \n",
    "from openai import OpenAI\n",
    "import pickle\n",
    "from dotenv import dotenv_values\n",
    "import pandas as pd\n",
    "import tiktoken\n",
    "import numpy as np\n",
    "from sklearn.manifold import TSNE\n",
    "import plotly.graph_objects as go\n",
    "from plotly.offline import plot\n",
    "from IPython.display import display, HTML\n",
    "import plotly.io as pio\n",
    "pio.renderers.default = \"plotly_mimetype\"\n",
    "\n",
    "config = dotenv_values(\"../.env\")\n",
    "\n",
    "DATA_ROOT = \"../data/repro-mri-scoping/repro_mri_scoping\"\n",
    "\n",
    "try:\n",
    "    client = OpenAI(api_key=config['OPENAI_KEY'])\n",
    "except:\n",
    "    config = {}\n",
    "    config['OPENAI_KEY'] = False\n",
    "    \n",
    "\n",
    "def read_file(file_path):\n",
    "    with open(file_path, 'r', encoding='utf-8') as file:\n",
    "        content = file.read()\n",
    "    return content\n",
    "\n",
    "def write_file(file_path, content):\n",
    "    with open(file_path, 'w') as file:\n",
    "        file.write(content)\n",
    "\n",
    "def write_pickle(filename,content):\n",
    "    with open(filename, 'wb') as file:\n",
    "        pickle.dump(content, file)\n",
    "\n",
    "def read_pickle(file_name):\n",
    "    with open(file_name, 'rb') as f:\n",
    "        data = pickle.load(f)\n",
    "    return data\n",
    "\n",
    "def get_output_dir(file_name):\n",
    "    op_dir = \"../output\"\n",
    "    if not os.path.exists(op_dir):\n",
    "        os.mkdir(op_dir)\n",
    "    return os.path.join(op_dir,file_name)\n",
    "\n",
    "def get_embedding(text, model=\"text-embedding-ada-002\"):\n",
    "    text = text.replace(\"\\n\", \" \")\n",
    "    return client.embeddings.create(input = [text], model=model).data[0].embedding"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8797ec42-e95c-491b-b4ab-c803c3cec1ff",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "```{warning} Setting up OpenAI authorization\n",
    "To enable API calls, you need to add your OpenAI token to the `.env` file.\n",
    "```\n",
    "\n",
    "```{warning} Starting a session with OpenAI API\n",
    "The following code requires GPT tokens to feed insight files one by one and to return a summary. Estimated cost ~6-8 USD.\n",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b65bed0a-9a5e-4f19-ab1b-e56a9fb27847",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": [
     "hide-input"
    ]
   },
   "outputs": [],
   "source": [
    "#| label: gptsession\n",
    "\n",
    "directory_path = os.path.join(DATA_ROOT,\"repro_insights_parsed_nov23\")\n",
    "\n",
    "# Read the data that has been crawled and parsed.\n",
    "input_files = [os.path.join(directory_path,f) for f in os.listdir(directory_path) if f.endswith('.txt')]\n",
    "\n",
    "# Task definition for GPT\n",
    "messages = [{\"role\": \"system\", \"content\": \"You are a helpful assistant designed to understand and summarize the reproducibility aspects of neuroimaging research articles.\"}]\n",
    "\n",
    "\"\"\"\n",
    "!!!! WARNING !!!!\n",
    "\n",
    "The following code requires GPT tokens to \n",
    "feed insight files one by one and to return \n",
    "a summary. Estimated cost ~6-8 USD.\n",
    "\"\"\"\n",
    "\n",
    "if config['OPENAI_KEY']:\n",
    "    # Comment out this line to run it on all the entries.\n",
    "    input_files = input_files[0:2]\n",
    "    \n",
    "    for ii, cur_file in enumerate(input_files):\n",
    "        cur_content = read_file(cur_file)\n",
    "        print(f\"Requesting summary for {cur_file}\")\n",
    "        messages.append({\"role\": \"user\", \"content\": f\"The following content is from ARTICLE {ii+1}. In relavance to its title and abstract, create 4 keywords, then summarize what did the study achieve and why it was reproducible in 4 sentences: {cur_content}\"})\n",
    "        response = client.chat.completions.create(\n",
    "          model=\"gpt-4-1106-preview\",\n",
    "          messages= messages\n",
    "        )\n",
    "        try:\n",
    "            cur_response = {\"role\": \"assistant\", \"content\": response.choices[0].message.content}\n",
    "            messages.append(cur_response)\n",
    "            write_pickle(get_output_dir(\"gpt_chat.pkl\"),messages)\n",
    "        except Exception as e:        \n",
    "            print(f\"Exception: {e}\")\n",
    "            write_pickle(get_output_dir(\"gpt_chat.pkl\"),messages)\n",
    "    gpt_outputs = os.path.join(get_output_dir(\"gpt_chat.pkl\"))            \n",
    "else: \n",
    "    print('API key not found. Available dataset will be used. Example: \\n')\n",
    "    gpt_outputs = os.path.join(DATA_ROOT, \"gpt_chat.pkl\")\n",
    "    messages = read_pickle(gpt_outputs)\n",
    "    print(messages[0:3])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a208a94f-c322-4c5f-b165-b6f61e93c0d2",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "```{admonition} Synthesis\n",
    "Create a high-level review of all the summarized content\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b499ca46-e9a5-4d71-8389-7d31b5c8723f",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": [
     "hide-input"
    ]
   },
   "outputs": [],
   "source": [
    "# List of dictionaries\n",
    "if config['OPENAI_KEY']:\n",
    "    # THIS ASSUMES THAT THE OUTPUT WAS MADE AVAILABLE IN THE PREVIOUS CELL    \n",
    "    messages.append({\"role\": \"user\", \"content\": f\"Generate a comprehensive scoping review on the reproducibility of neuroimaging, incorporating insights from the 31 articles you summarized in our previous conversation. Provide examples and key findings to offer a thorough examination of the current state of reproducibility in neuroimaging research.\"})\n",
    "    \n",
    "    response = client.chat.completions.create(\n",
    "       model=\"gpt-4-1106-preview\",\n",
    "       messages= messages)\n",
    "    \n",
    "    cur_response = {\"role\": \"assistant\", \"content\": response.choices[0].message.content}\n",
    "    messages.append(cur_response)\n",
    "    \n",
    "    article = messages[-1]['content']\n",
    "    \n",
    "    with open(get_output_dir('review_article_summary.txt'), 'w') as file:\n",
    "        file.write(article)\n",
    "    \n",
    "    print(messages[-1]['content'])\n",
    "else:\n",
    "    # Simply print the output\n",
    "    print('API key not found. An example output will be displayed from the previous results. \\n')\n",
    "    gpt_txt = read_file(os.path.join(DATA_ROOT, \"review1.txt\"))\n",
    "    print(gpt_txt)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a11f53e0-9c53-4a87-b4e5-ef7527a70ab1",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "```{admonition} Save outputs\n",
    "Save GPT-generated summaries with matching paper IDs\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "433bffa7-ea80-4ac0-aa23-51b0482528e0",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": [
     "hide-input"
    ]
   },
   "outputs": [],
   "source": [
    "if config['OPENAI_KEY']:\n",
    "\n",
    "    # messages variable should be in scope. Please see previous cells.\n",
    "    # Fetch this from the saved data\n",
    "    input_files = [os.path.join(DATA_ROOT, \"gpt_insights_parsed_nov23\", f) for f in os.listdir(directory_path) if f.endswith('.txt')]\n",
    "    \n",
    "    if not os.path.exists(get_output_dir(\"summary\")):\n",
    "        os.mkdir(get_output_dir(\"summary\"))\n",
    "    \n",
    "    summary_files = [os.path.join(get_output_dir(\"summary\"), f) for f in os.listdir(directory_path) if f.endswith('.txt')]\n",
    "    \n",
    "    large_summary = ''\n",
    "    it = 0\n",
    "    for msg in messages:\n",
    "        if msg['role'] == \"assistant\":\n",
    "            large_summary += \"\\n\" + msg['content'] + \"\\n ---------------------------- \\n\"\n",
    "            write_file(summary_files[it],msg['content'])\n",
    "            it = it +1 \n",
    "    \n",
    "    write_file(get_output_dir('gpt_combined_summaries.txt'),large_summary)\n",
    "else:\n",
    "    print('API key not found. 3 example outputs will be displayed. \\n')\n",
    "    gpt_outputs = os.path.join(DATA_ROOT, \"gpt_chat.pkl\")\n",
    "    messages = read_pickle(gpt_outputs)\n",
    "    it = 0\n",
    "    for msg in messages:\n",
    "        if msg['role'] == \"assistant\":\n",
    "           print(msg['content'] + '\\n')\n",
    "           it+=1\n",
    "        if it == 3:\n",
    "            break\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9b18b8f-435e-4e0b-b8a2-18d8cf4f4cdb",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "```{admonition} Create GPT embeddings\n",
    "Use `text-embedding-ada-002` model to create embeddings for the summaries.\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a4504bb8-1deb-4a62-85a4-5b50f00ecf6c",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": [
     "hide-input"
    ]
   },
   "outputs": [],
   "source": [
    "if config['OPENAI_KEY']:\n",
    "    # Output path will be checked for the summaries \n",
    "    summary_files = [os.path.join(get_output_dir(\"summary\", f)) for f in os.listdir(directory_path) if f.endswith('.txt')]\n",
    "    \n",
    "    # List to store summaries\n",
    "    summaries = []\n",
    "    \n",
    "    # Iterate through each txt file in the directory\n",
    "    for filename in summary_files:\n",
    "        content = read_file(filename)\n",
    "        summaries.append(content)\n",
    "    \n",
    "    # Create a Pandas DataFrame with a column named 'summary'\n",
    "    df = pd.DataFrame({'summary': summaries})\n",
    "    \n",
    "    embedding_model = \"text-embedding-ada-002\"\n",
    "    embedding_encoding = \"cl100k_base\"  # this the encoding for text-embedding-ada-002\n",
    "    max_tokens = 8000\n",
    "    top_n = 1000\n",
    "    \n",
    "    encoding = tiktoken.get_encoding(embedding_encoding)\n",
    "    \n",
    "    # omit reviews that are too long to embed\n",
    "    df[\"n_tokens\"] = df.summary.apply(lambda x: len(encoding.encode(x)))\n",
    "    df = df[df.n_tokens <= max_tokens].tail(top_n)\n",
    "\n",
    "    # Call GPT for embedding\n",
    "    df[\"embedding\"] = df.summary.apply(lambda x: get_embedding(x, model='text-embedding-ada-002'))\n",
    "    df.to_csv(get_output_dir('gpt_summary_embedding.csv'), index=False)\n",
    "else:\n",
    "    # Read embedding data that has been saved\n",
    "    df = pd.read_csv(os.path.join(DATA_ROOT,'gpt_summary_embedding_nov23.csv'))\n",
    "    print(df.head())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1d7f336a-e239-4fbd-9c45-ba97b7351a8d",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": [
     "hide-input"
    ]
   },
   "outputs": [],
   "source": [
    "#| label: gptembedding\n",
    "# Get embeddings as matrix from the dataframe\n",
    "matrix = df.embedding.apply(eval).to_list()\n",
    "\n",
    "# Use T-distributed Stochastic Neighbor Embedding to create joint probabilities.\n",
    "tsne = TSNE(n_components=2, perplexity=15, random_state=42, init='random', learning_rate=200)\n",
    "vis_dims = tsne.fit_transform(np.array(matrix))\n",
    "\n",
    "x = [x for x,y in vis_dims]\n",
    "y = [y for x,y in vis_dims]\n",
    "\n",
    "fig = go.Figure()\n",
    "\n",
    "# Scatter plot for UMAP in 2D\n",
    "scatter_2d = go.Scatter(\n",
    "    x=x,\n",
    "    y=y,\n",
    "    mode='markers',\n",
    "    # Color w.r.t. summary length\n",
    "    marker = dict(color =[len(sm) for sm in df.summary],\n",
    "                                          size=9, \n",
    "                                          opacity=0.9,\n",
    "                                          colorbar=dict(thickness=10,x=1.1, title=\"N chars\")),\n",
    "    customdata= [sm.split('\\n')[0] for sm in df.summary],\n",
    "    hovertemplate='%{customdata}',\n",
    "    visible = True,\n",
    "    name=''\n",
    ")\n",
    "fig.add_trace(scatter_2d)\n",
    "\n",
    "fig.update_layout(yaxis={'visible': False, 'showticklabels': False})\n",
    "fig.update_layout(xaxis={'visible': False, 'showticklabels': False})\n",
    "\n",
    "# Update layout\n",
    "fig.update_layout(title = 't-SNE based visualization of the summary embeddings',\n",
    "                 height = 800,\n",
    "                 width = 800,\n",
    "                 template = 'plotly_dark',\n",
    "                 hovermode='closest',\n",
    "                 showlegend = True)\n",
    "\n",
    "\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4d775616-adc2-4c95-b68b-7951d955e1ec",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import requests\n",
    "\n",
    "DATA_ROOT = \"../data/repro-mri-scoping/repro_mri_scoping/gpt_summary_nov23\"\n",
    "\n",
    "# Function to make Semantic Scholar API call and extract title and authors\n",
    "def get_paper_info(paper_id):\n",
    "    api_url = f\"https://api.semanticscholar.org/graph/v1/paper/{paper_id}?fields=url,year,authors,title\"\n",
    "    response = requests.get(api_url)\n",
    "    \n",
    "    if response.status_code == 200:\n",
    "        data = response.json()\n",
    "        title = data.get('title', 'Title Not Available')\n",
    "        year = data.get('year', 'Year Not Available')\n",
    "        authors = ', '.join(author.get('name', 'Unknown Author') for author in data.get('authors', []))\n",
    "        return title, authors, year\n",
    "    else:\n",
    "        print(f\"Failed to fetch data for paper_id: {paper_id}\")\n",
    "        return None, None, None\n",
    "\n",
    "# Directory containing txt files\n",
    "directory = DATA_ROOT\n",
    "\n",
    "# Initialize an empty list to store data\n",
    "combined_data = []\n",
    "\n",
    "# Loop through each txt file in the directory\n",
    "it =1\n",
    "for filename in os.listdir(directory):\n",
    "    if filename.endswith(\".txt\"):\n",
    "        paper_id = os.path.splitext(filename)[0]\n",
    "        file_path = os.path.join(directory, filename)\n",
    "        \n",
    "        # Read content of the txt file\n",
    "        with open(file_path, 'r', encoding='utf-8') as file:\n",
    "            content = file.read().strip()\n",
    "        \n",
    "        # Make Semantic Scholar API call to get paper title and authors\n",
    "        title, authors, year = get_paper_info(paper_id)\n",
    "        \n",
    "        # Append data to the list\n",
    "        combined_data.append(f\"\\nPaper number: {it}\\nYear: {year}\\nTitle: {title}\\nAuthors: {authors}\\n\\n{content}\\n\\n ---------------------------------------------------\")\n",
    "        it = it+1\n",
    "# Write the combined data to a text file\n",
    "with open(os.path.join(get_output_dir('combined_papers_insight.txt')), 'w', encoding='utf-8') as output_file:\n",
    "    output_file.writelines(combined_data)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "768b3de2-9274-497e-be86-2cdae19cb342",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "```{bibliography}\n",
    "```"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}