{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Single-cell Profiling Walkthrough\n",
    "\n",
    "Welcome to this walkthrough where we will guide you through the process of extracting single cell morphology features using the [Pycytominer](https://github.com/cytomining/pycytominer) API.\n",
    "\n",
    "For this walkthrough, we will be working with the NF1-Schwann cell morphology dataset. \n",
    "If you would like the more information about this dataset, you can refer to this [repository](https://github.com/WayScience/nf1_cellpainting_data)\n",
    "\n",
    "From the mentioned repo, we specifically used this [dataset](https://github.com/WayScience/nf1_cellpainting_data/tree/main/2.cellprofiler_analysis/analysis_output) and the associated [metadata](https://github.com/WayScience/nf1_cellpainting_data/tree/main/0.download_data/metadata) to generate the walkthrough. \n",
    "\n",
    "\n",
    "Let's get started with the walkthrough below!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pathlib\n",
    "\n",
    "# ignore mix type warnings from pandas\n",
    "import warnings\n",
    "\n",
    "import pandas as pd\n",
    "\n",
    "from pycytominer import annotate, feature_select, normalize\n",
    "\n",
    "# pycytominer imports\n",
    "from pycytominer.cyto_utils.cells import SingleCells\n",
    "\n",
    "warnings.filterwarnings(\"ignore\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## About the inputs\n",
    "\n",
    "\n",
    "In this section, we will set up the expected input and output paths that will be generated throughout this walkthrough. Let's take a look at the explanation of these inputs and outputs.\n",
    "\n",
    "For this workflow, we have two main inputs:\n",
    "\n",
    "- **plate_data** (SQLite file): This contains the quantified single-cell morphology features that we'll be working with.\n",
    "- **plate_map** (CSV file): This contains additional information related to the cells, providing valuable context of our single-cell morphology dataset.\n",
    "\n",
    "Now, let's explore the outputs generated in this workflow. In this walkthrough, we will be generating four profiles:\n",
    "\n",
    "- **sc_profiles_path**: This refers to the single-cell morphology profile that will be generated.\n",
    "- **anno_profiles_path**: This corresponds to the annotated single-cell morphology profile, meaning the information from the **plate_map** is added to each single-cell\n",
    "- **norm_profiles_path**: This represents the normalized single-cell morphology profile.\n",
    "- **feat_profiles_path**: Lastly, this refers to the selected features from the single-cell morphology profile.\n",
    "\n",
    "**Note**: All profiles are outputted as `.csv` files, however, users can output these files as `parquet` or other file file formats. \n",
    "For more information about output formats, please refer the documentation [here](https://pycytominer.readthedocs.io/en/latest/pycytominer.cyto_utils.html#pycytominer.cyto_utils.output.output) \n",
    "\n",
    "These profiles will serve as important outputs that will help us analyze and interpret the single-cell morphology data effectively. Now that we have a clear understanding of the inputs and outputs, let's proceed further in our walkthrough."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Setting file paths\n",
    "data_dir = pathlib.Path(\"./data/\").resolve(strict=True)\n",
    "metadata_dir = (data_dir / \"metadata\").resolve(strict=True)\n",
    "out_dir = pathlib.Path(\"results\")\n",
    "out_dir.mkdir(exist_ok=True)\n",
    "\n",
    "# input file paths\n",
    "plate_data = pathlib.Path(\"./data/nf1_data.sqlite\").resolve(strict=True)\n",
    "plate_map = (metadata_dir / \"platemap_NF1_CP.csv\").resolve(strict=True)\n",
    "\n",
    "# setting output paths\n",
    "sc_profiles_path = out_dir / \"nf1_single_cell_profile.csv.gz\"\n",
    "anno_profiles_path = out_dir / \"nf1_annotated_profile.csv.gz\"\n",
    "norm_profiles_path = out_dir / \"nf1_normalized_profile.csv.gz\"\n",
    "feat_profiles_path = out_dir / \"nf1_features_profile.csv.gz\""
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generating Merged Single-cell Morphology Dataset\n",
    "\n",
    "In this section of the walkthrough, our goal is to load the NF1 dataset and create a merged single-cell morphology dataset.\n",
    "\n",
    "Currently, the NF1 morphology `SQLite` dataset was generated by using [CellProfiler's](https://github.com/CellProfiler/CellProfiler) [ExportToDatabase](https://cellprofiler-manual.s3.amazonaws.com/CellProfiler-4.2.5/modules/fileprocessing.html?highlight=exporttodatabase#module-cellprofiler.modules.exporttodatabase) function, where each table represents a different compartment, such as Image, Cell, Nucleus, and Cytoplasm.\n",
    "\n",
    "To achieve this, we will utilize the `SingleCells` class, which offers a range of functionalities specifically designed for single-cell analysis. You can find detailed documentation on these functionalities [here](https://pycytominer.readthedocs.io/en/latest/pycytominer.cyto_utils.html#pycytominer.cyto_utils.cells.SingleCells).\n",
    "\n",
    "However, for our purpose in this walkthrough, we will focus on using the `SingleCells` class to merge all the tables within the NF1 sqlite file into a merged single-cell morphology dataset.\n",
    "\n",
    "### Updating defaults\n",
    "Before we proceed further, it is important to update the default parameters in the `SingleCells`class to accommodate the table name changes in our NF1 dataset.\n",
    "\n",
    "Since the table names in our NF1 dataset differ from the default table names recognized by the `SingleCells` class, we need to make adjustments to ensure proper recognition of these table name changes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# update compartment names and strata\n",
    "strata = [\"Image_Metadata_Well\", \"Image_Metadata_Plate\"]\n",
    "compartments = [\"Per_Cells\", \"Per_Cytoplasm\", \"Per_Nuclei\"]\n",
    "\n",
    "# Updating linking columns for merging all compartments\n",
    "linking_cols = {\n",
    "    \"Per_Cytoplasm\": {\n",
    "        \"Per_Cells\": \"Cytoplasm_Parent_Cells\",\n",
    "        \"Per_Nuclei\": \"Cytoplasm_Parent_Nuclei\",\n",
    "    },\n",
    "    \"Per_Cells\": {\"Per_Cytoplasm\": \"Cells_Number_Object_Number\"},\n",
    "    \"Per_Nuclei\": {\"Per_Cytoplasm\": \"Nuclei_Number_Object_Number\"},\n",
    "}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have stored the updated parameters, we can use them as inputs for `SingleCells` class to merge all the NF1 sqlite tables into a single consolidated dataset.\n",
    "\n",
    "This is done through the `merge_single_cells` method. For more infromation about `merge_single_cells` please refer to the documentation [here](https://pycytominer.readthedocs.io/en/latest/pycytominer.cyto_utils.html#pycytominer.cyto_utils.cells.SingleCells.merge_single_cells)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PosixPath('results/nf1_single_cell_profile.csv.gz')"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# setting up sqlite address\n",
    "sqlite_address = f\"sqlite:///{plate_data!s}\"\n",
    "\n",
    "# loading single cell morphology data into pycyotminer's SingleCells Object\n",
    "single_cell_profile = SingleCells(\n",
    "    sql_file=sqlite_address,\n",
    "    compartments=compartments,\n",
    "    compartment_linking_cols=linking_cols,\n",
    "    image_table_name=\"Per_Image\",\n",
    "    strata=strata,\n",
    "    merge_cols=[\"ImageNumber\"],\n",
    "    image_cols=\"ImageNumber\",\n",
    "    load_image_data=True,\n",
    ")\n",
    "\n",
    "# merging all sqlite table into a single tabular dataset (csv) and save as\n",
    "# compressed csv file\n",
    "single_cell_profile.merge_single_cells(\n",
    "    sc_output_file=sc_profiles_path, compression_options=\"gzip\"\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have created our merged single-cell profile, let's move on to the next step: loading our `platemaps`. \n",
    "\n",
    "`Platemaps` provide us with additional information that is crucial for our analysis. They contain details such as well positions, genotypes, gene names, perturbation types, and more. In other words, platemaps serve as a valuable source of metadata for our single-cell morphology profile.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['WellRow', 'WellCol', 'well_position', 'gene_name', 'genotype']\n"
     ]
    }
   ],
   "source": [
    "# loading plate map and display it\n",
    "platemap_df = pd.read_csv(plate_map)\n",
    "\n",
    "# displaying platemap\n",
    "print(platemap_df.columns.tolist())"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Annotation\n",
    "\n",
    "In this step of the walkthrough, we will combine the metadata with the merged single-cell morphology dataset. To accomplish this, we will utilize the `annotation` function provided by `pycytominer`.\n",
    "\n",
    "The `annotation` function takes two inputs: the merged single-cell morphology dataset and its associated plate map. By combining these two datasets, we will generate an annotated_profile that contains enriched information.\n",
    "\n",
    "More information about the `annotation` function can be found [here](https://pycytominer.readthedocs.io/en/latest/pycytominer.html#module-pycytominer.annotate)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Annotated profile saved in: results/nf1_annotated_profile.csv.gz\n"
     ]
    }
   ],
   "source": [
    "# annotating merged single-cell profile with metadata\n",
    "annotate(\n",
    "    profiles=sc_profiles_path,\n",
    "    platemap=platemap_df,\n",
    "    join_on=[\"Metadata_well_position\", \"Image_Metadata_Well\"],\n",
    "    output_file=anno_profiles_path,\n",
    "    compression_options=\"gzip\",\n",
    ")\n",
    "\n",
    "# save message display\n",
    "print(f\"Annotated profile saved in: {anno_profiles_path}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Normalization Step\n",
    "\n",
    "The next step is to normalize our dataset using the `normalize` function provided by `pycytominer`.\n",
    "More information regards `pycytominer`'s `normalize` function can be found [here](https://pycytominer.readthedocs.io/en/latest/pycytominer.html#module-pycytominer.normalize)\n",
    "\n",
    "Normalization is a critical preprocessing step that improves the quality of our dataset. It addresses two key challenges: mitigating the impact of outliers and handling variations in value scales. By normalizing the data, we ensure that our downstream analysis is not heavily influenced by these factors.\n",
    "\n",
    "Additionally, normalization plays a crucial role in determining feature importance (which is crucial for our last step). By bringing all features to a comparable scale, it enables the identification of important features without biases caused by outliers or widely-scaled values.\n",
    "\n",
    "To normalize our annotated single-cell morphology profile, we will utilize the `normalize` function from `pycytominer`. This function is specifically designed to handle the normalization process for cytometry data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Normalized profile saved in: results/nf1_normalized_profile.csv.gz\n"
     ]
    }
   ],
   "source": [
    "# normalize dataset\n",
    "normalize(\n",
    "    profiles=anno_profiles_path,\n",
    "    features=\"infer\",\n",
    "    image_features=False,\n",
    "    meta_features=\"infer\",\n",
    "    samples=\"all\",\n",
    "    method=\"standardize\",\n",
    "    output_file=norm_profiles_path,\n",
    "    compression_options=\"gzip\",\n",
    ")\n",
    "\n",
    "# save message display\n",
    "print(f\"Normalized profile saved in: {norm_profiles_path}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature Selection\n",
    "\n",
    "\n",
    "In the final section of our walkthrough, we will utilize the normalized dataset to extract important morphological features and generate a selected features profile.\n",
    "\n",
    "To accomplish this, we will make use of the `feature_select` function provided by `pycytominer`. \n",
    "Using `pycytominer`'s `feature_select` function to our dataset, we can identify the most informative morphological features that contribute significantly to the variations observed in our data. These selected features will be utilized to create our feature profile.\n",
    "\n",
    "For more detailed information about the `feature_select` function, its parameters, and its capabilities, please refer to the documentation available [here](https://pycytominer.readthedocs.io/en/latest/pycytominer.html#module-pycytominer.feature_select).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Selected features profile saved in: results/nf1_features_profile.csv.gz\n"
     ]
    }
   ],
   "source": [
    "# creating selected features profile\n",
    "feature_select(\n",
    "    profiles=norm_profiles_path,\n",
    "    features=\"infer\",\n",
    "    image_features=False,\n",
    "    samples=\"all\",\n",
    "    operation=[\"variance_threshold\", \"correlation_threshold\", \"blocklist\"],\n",
    "    output_file=feat_profiles_path,\n",
    "    compression_options=\"gzip\",\n",
    ")\n",
    "\n",
    "# save message display\n",
    "print(f\"Selected features profile saved in: {feat_profiles_path}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Congratulations! You have successfully completed our walkthrough. We hope that it has provided you with a basic understanding of how to analyze cell morphology features using `pycytominer`.\n",
    "\n",
    "By following the steps outlined in this walkthrough, you have gained valuable insights into processing high-dimensional single-cell morphology data with ease using `pycytominer`. However, please keep in mind that `pycytominer` offers a wide range of functionalities beyond what we covered here. We encourage you to explore the documentation to discover more advanced features and techniques.\n",
    "\n",
    "If you have any questions or need further assistance, don't hesitate to visit the `pycytominer` repository and post your question in the [issues](https://github.com/cytomining/pycytominer/issues) section. The community is there to support you and provide guidance.\n",
    "\n",
    "Now that you have the knowledge and tools to analyze cell morphology features, have fun exploring and mining your data!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "pycytominer",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}