Skip to content

Commit

Permalink
test push readme, devcont, installer
Browse files Browse the repository at this point in the history
  • Loading branch information
sirbastiano committed Mar 23, 2024
1 parent 0e2da5e commit 3e79e38
Show file tree
Hide file tree
Showing 7 changed files with 536 additions and 526 deletions.
20 changes: 20 additions & 0 deletions .devcontainer/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
############################################# INSTRUCTIONS #############################################
# build image: docker build -t pyraws:latest --build-arg CACHEBUST=$(date +%s) -f dockerfile .
# run image: docker run -it --rm -p 8888:8888 pyraws:latest
############################################# END #############################################

# Use the official PyTorch base image
FROM pytorch/pytorch:1.11.0-cuda11.3-cudnn8-runtime

# Set the working directory inside the container
WORKDIR /workspace

# Install Python dependencies for pyraws:
RUN apt-get update && apt-get install -y git ffmpeg libsm6 libxext6
# install pyraws

RUN git clone -b main https://github.com/ESA-PhiLab/PyRawS.git

RUN cd PyRawS && /bin/bash -c "source ./pyraws_install.sh"
RUN echo "source activate pyraws" > ~/.bashrc
ENV PATH /opt/conda/envs/pyraws/bin:$PATH
22 changes: 22 additions & 0 deletions .devcontainer/devontainer.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
// For format details, see https://aka.ms/devcontainer.json. For config options, see the
// README at: https://github.com/devcontainers/templates/tree/main/src/anaconda

{
"name": "pyraws",
"dockerComposeFile": "docker-compose.yml",
"service": "pyraws",
"workspaceFolder": "/workspace",
"settings": {
"terminal.integrated.shell.linux": "/bin/bash"
},
"extensions": [
"ms-toolsai.jupyter",
"ms-toolsai.vscode-jupyter-cell-tags",
"ms-toolsai.jupyter-renderers",
"ms-toolsai.vscode-jupyter-powertoys",
"ms-toolsai.vscode-jupyter-slideshow",
"donjayamanne.python-extension-pack"
],
// Use 'postCreateCommand' to run commands after the container is created.
//"postCreateCommand": "conda install -n simulator ipykernel --update-deps --force-reinstall"
}
20 changes: 20 additions & 0 deletions .devcontainer/docker-compose.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
version: "3.8"

services:
pyraws:
image: pyraws:latest
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: 1
capabilities: [gpu]
build:
context: .
dockerfile: Dockerfile
volumes:
- ../:/workspace/
platform: linux/amd64

command: sleep infinity
481 changes: 13 additions & 468 deletions README.md

Large diffs are not rendered by default.

34 changes: 34 additions & 0 deletions docs/GLOSSARY.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@

## Glossary
* ### Coarse coregistration
Lightweight spatial coregistration method optimized for onboard-satellite applications. It simply shifts the various bands of a fixed factor that depends only on the bands, the satellite and detector number.

* ### Sentinel-2 L0 data
Sentinel-2 data at `level-0` (`L0`) are data that are transmitted to Ground from Sentinel-2 satellites. The `L0` format is compressed to diminish downlink bandwidth requirements. For more information, refer to the [Sentinel-2 Products Specification Document](https://sentinel.esa.int/documents/247904/685211/sentinel-2-products-specification-document)

* ### Sentinel-2 Raw data
In the frame of this project, the [Sentinel-2 Raw](#sentinel-2-raw-data) represents a particular product in the Sentinel-2 processing chain that matches a decompressed version of [Sentinel-2 L0 data](sentinel-2-l0-data) with additional metadata that are produced on ground. Once decompressed, `Sentinel-2 Raw data` are the data available on Ground that better emulate the one produced by Sentinel-2 detectors with the exception of the effects due to compression and onboard equalization, which are not compensated at this stage. Therefore, `Sentinel-2 raw data` are those exploited in this project. For more information, refer to the [Sentinel-2 Products Specification Document](https://sentinel.esa.int/documents/247904/685211/sentinel-2-products-specification-document). <br>
**N.B**: the nomenclature ```raw data``` and its location in the Sentinel-2 processing chain is specific for this project only.

* ### Sentinel-2 Raw granule
A `granule` is the image acquired by a Sentinel-2 detector during a single acquisition lasting 3.6 s. Granules are defined at [L0](#sentinel-2-raw-data) level. However, since the processing perfomed on the ground between L0 and raw data does not alter the image content (with the exception of the decompression process) but just provide additional metadata, granules are defined also at [Sentinel-2 Raw](#sentinel-2-raw-data) level.
Given the pushbroom nature of the Sentinel-2 sensor, bands do not look at the same area at [Raw](#sentinel-2-raw-data) level. For more information, refer to the [Sentinel-2 Products Specification Document](https://sentinel.esa.int/documents/247904/685211/sentinel-2-products-specification-document)

* ### Sentinel-2 Raw event
[Sentinel-2 Raw data](#sentinel-2-raw-data) are produced by decompressing [Sentinel-2 L0 data](sentinel-2-raw-data). To download L0 data, it is necessary to specify one polygon that surrounds a particular area-of-interest. This leads to download all those [Sentinel-2 Raw granules](#sentinel-2-raw-granule) whose reference band intersects the specified polygon. Such collection is a `Raw-event`. Each `Raw-event` matches one of the `ID_event` entry of the database. <br>
For each `Raw-event`, we do not provide all the collection of [Sentinel-2 Raw granules](#sentinel-2-Raw-granule), but only the set of [Raw data useful granules](#raw-data-useful-granule) and [Raw data complementary granules](#Raw-data-complementary-granule). For an intuitive example, please, check [Raw events and granules](#raw-events-and-raw-granules).

* ### Sentinel-2 L1C data
The `Level 1-C` (`L1C`) is one format for `Sentinel-2` data. To convert [Sentinel-2 Raw data](#sentinel-2-raw-data) to `L1C` data, numerous processing steps are applied to correct defects, including bands coregistration, ortho-rectification, decompression, noise-suppression and other. For more information, refer to the [Sentinel-2 Products Specification Document](https://sentinel.esa.int/documents/247904/685211/sentinel-2-products-specification-document).

* ### Sentinel-2 L1C event
Same concept for [Sentinel-2 Raw events](#sentinel-2-raw-event) but applied on [Sentinel-2 L1C data](#sentinel-2-l1c-data).

* ### Sentinel-2 L1C tile
The `Sentinel-2 L1C tile` is the minimum `L1C` product that can be downloaded.

* ### Raw complementary granule
Given a certain set of bands of interest `[Bx,By,...,Bz]`, `Raw complementarey granules` are the granules adjacents at [Raw-useful-granules](#raw-useful-granule) that that can be used to fill missing pixels of `[By,...,Bz]` bands due to their coregistration with respecto the band `Bx`. For an intuitive example, please, check [Raw events and granules](#raw-events-and-raw-granules).

* ### Raw useful granule
Given a certain set of bands of interest `[Bx,By,...,Bz]`, where `Bx` is the first band in the set, an `Raw useful granule` is one of the collection of [Sentinel-2 Raw granules](#sentinel-2-raw-granule) that compose a [Sentinel-2 Raw event](#sentinel-2-raw-event) whose band `Bx` include (or intersects) a certain area of interest (e.g., an eruption or an area covered by a fire). For an intuitive example, please, check [Raw data events and granules](#raw-events-and-raw-granules).
99 changes: 41 additions & 58 deletions pyraws_install.sh
Original file line number Diff line number Diff line change
Expand Up @@ -9,10 +9,10 @@ then
exit 1
fi

# Create a new Conda environment called openmmlab
# Create a new Conda environment called pyraws
conda create --name pyraws python=3.9 -y

# Activate the AINavi environment
# Activate the environment
source $(conda info --base)/etc/profile.d/conda.sh
if conda activate pyraws; then
echo "pyraws environment activated"
Expand All @@ -21,62 +21,45 @@ else
exit 1
fi

#!/bin/bash
# Check the operating system and install pytorch accordingly
if [[ "$OSTYPE" == "darwin"* ]]; then
# macOS
echo "Detected macOS"
conda install -y pytorch==1.11.0 torchvision==0.12.0 torchaudio==0.11.0 -c pytorch
elif [[ "$OSTYPE" == "linux-gnu" ]]; then
# Linux
echo "Detected Linux"
if command -v nvcc >/dev/null 2>&1; then
# CUDA is available
if nvcc --version | grep "release 10\.2" >/dev/null 2>&1; then
# CUDA 10.2
echo "Detected CUDA 10.2"
conda install -y pytorch==1.11.0 torchvision==0.12.0 torchaudio==0.11.0 cudatoolkit=10.2 -c pytorch
elif nvcc --version | grep "release 11\.3" >/dev/null 2>&1; then
# CUDA 11.3
echo "Detected CUDA 11.3"
conda install -y pytorch==1.11.0 torchvision==0.12.0 torchaudio==0.11.0 cudatoolkit=11.3 -c pytorch
else
# CUDA version not supported, installing CPU version
echo "CUDA version not supported, installing CPU version"
conda install -y pytorch==1.11.0 torchvision==0.12.0 torchaudio==0.11.0 cpuonly -c pytorch
fi
else
# CUDA is not available, installing CPU version
echo "CUDA not detected, installing CPU version"
conda install -y pytorch==1.11.0 torchvision==0.12.0 torchaudio==0.11.0 cpuonly -c pytorch
fi
elif [[ "$OSTYPE" == "msys" ]]; then
# Windows
echo "Detected Windows"
if where nvcc >/dev/null 2>&1; then
# CUDA is available
if nvcc --version | grep "release 10\.2" >/dev/null 2>&1; then
# CUDA 10.2
echo "Detected CUDA 10.2"
conda install -y pytorch==1.11.0 torchvision==0.12.0 torchaudio==0.11.0 cudatoolkit=10.2 -c pytorch
elif nvcc --version | grep "release 11\.3" >/dev/null 2>&1; then
# CUDA 11.3
echo "Detected CUDA 11.3"
conda install -y pytorch==1.11.0 torchvision==0.12.0 torchaudio==0.11.0 cudatoolkit=11.3 -c pytorch
else
# CUDA version not supported, installing CPU version
echo "CUDA version not supported, installing CPU version"
conda install -y pytorch==1.11.0 torchvision==0.12.0 torchaudio==0.11.0 cpuonly -c pytorch
fi
else
# CUDA is not available, installing CPU version
echo "CUDA not detected, installing CPU version"
conda install -y pytorch==1.11.0 torchvision==0.12.0 torchaudio==0.11.0 cpuonly -c pytorch
fi
else
echo "Unsupported operating system"
exit 1
fi

# Define PyTorch version and components
pytorch_version="1.11.0"
torchvision_version="0.12.0"
torchaudio_version="0.11.0"

# Function to install PyTorch
install_pytorch() {
conda install -y pytorch==$pytorch_version torchvision==$torchvision_version torchaudio==$torchaudio_version $1 -c pytorch
}

# Install via pip:
# pip install torch==1.12.1+cu113 torchvision==0.13.1+cu113 torchaudio==0.12.1 --extra-index-url https://download.pytorch.org/whl/cu113

# Detect operating system
case "$OSTYPE" in
"darwin"*)
# macOS
echo "Detected macOS"
install_pytorch "cpuonly"
;;
"linux-gnu"|"linux-gnueabihf")
# Linux
echo "Detected Linux"
# Install PyTorch with CUDA 11.3 support
install_pytorch "cudatoolkit=11.3"
;;
"msys"|"win32")
# Windows
echo "Detected Windows"
# Install PyTorch with CUDA 11.3 support
install_pytorch "cudatoolkit=11.3"
;;
*)
echo "Unsupported operating system"
exit 1
;;
esac


# get absolute path of current working directory
# and setup the sys_cfg.py file
Expand Down
Loading

0 comments on commit 3e79e38

Please sign in to comment.