Facing issues in setting up environment for data science (R, Python)

General
1

Hi,
Was trying to setup Data Science environment in idx but couldn’t . Seems like idx can’t find rPackages. Any workaround?

Here’s my dev.nix config for reference :

# dev.nix configuration with correct R package
{ pkgs, ... }: {
  channel = "stable-24.05"; # Choose the appropriate nixpkgs channel
  packages = [
    pkgs.rPackages.tidyverse        # Equivalent to using rocker/tidyverse in Dockerfile
    pkgs.rPackages.iRkernel         # Jupyter kernel for R
    pkgs.python311Packages.jupyterlab                 # JupyterLab environment
    pkgs.rPackages.quarto                 # Quarto CLI for literate programming and markdown
    pkgs.python311                  # Python3 for JupyterLab, etc.
    pkgs.python311Packages.pip      # Python package installer
    pkgs.rstudio
    pkgs.rstudio-server
  ];
  
  # Set environment variables for the workspace
  env = {
    RSTUDIO_PORT = "8787";
    JUPYTER_PORT = "8888";
  };

  idx = {
    extensions = [
      # Equivalent VS Code extensions from open-vsx
      "REditorSupport.r"
      "ms-toolsai.jupyter"
      "ms-python.python"
      "ms-python.vscode-pylance"
      "GitHub.copilot"
      "vsls-contrib.codetour"
    ];

    # Workspace lifecycle hooks
    workspace = {
      # Runs when a workspace is first created
      onCreate = {
        # Install any Python dependencies mentioned in the requirements.txt file
        install-deps = "pip install --user -r requirements.txt";
        # Default open files like the dev.nix and README.md files
        default.openFiles = [ ".idx/dev.nix" "README.md" ];
      };

      # Runs when the workspace is (re)started
      onStart = {
        # Start RStudio Server when the workspace starts
        start-rstudio = "rstudio-server start";
        # Start JupyterLab without authentication
        start-jupyter = ''
          jupyter lab \
          --ServerApp.ip="0.0.0.0" \
          --ServerApp.allow_origin="*" \
          --ServerApp.port=8888 \
          --ServerApp.token="" \
          --ServerApp.password="" \
          --no-browser &
        '';
      };
    };
  };
}
2

Hey did you mean to specify IRkernel instead of iRkernel?

1 Like
3

My bad, sorry for the typo

environment.systemPackages = [
    pkgs.rPackages.IRkernel
  ];
4

Hello! I have been facing a similar issue for a package named ‘phidata’, i tried to install it via conda and run as well but still faced ModuleNotFoundError.

Did you happen to find a solution?

5

I was facing the same issue when I was trying to set up IDX for a data science/data analysis development environment.

The solution I found is to not use the IDX .nix package to install packages that I want to use, only use it to install the Python version and the corresponding pip version I want to use and then create a virtual environment and use poetry to install my packages inside of the virtual environments (you can just use pip).

Now, here is the detailed steps i asked ChatGPT to write :robot:

Step 1: Use the Provided .nix File

The first step is to use the provided .nix file to configure IDX. This file will ensure you have the correct Python version and pip installed for your project. Place the .nix file in your project root and let IDX handle the setup automatically.

The .nix file is provided at the end of this guide for reference.

Step 2: Activate Your Environment

To start using the virtual environment, you need to activate it in your terminal:

source .venv/bin/activate

If it worked, you’ll see (.venv) at the beginning of your terminal line. Now any pip install commands you run will only install packages inside this environment without affecting your system-wide Python setup.

Step 3: Make Sure You’re Using the Right Python

Next, check that your IDE (like VSCode) is using the Python interpreter inside your .venv folder. It’s usually located at .venv/bin/python. This ensures your code will use the packages installed in this virtual environment.

Step 4: Install Your Python Packages

You’re now ready to install the packages you need. For example, to install pandas, just run:

pip install pandas

Repeat this step for any other libraries your project requires.

Now, About Poetry

While pip is great for basic needs, managing dependencies can get messy as your project grows. That’s where Poetry comes in. It’s a powerful tool that handles dependencies, environments, and even project setup in one smooth package.

Here’s how to get started with Poetry:

  1. Install Poetry
    After activating your virtual environment, install Poetry using pip:

    pip install poetry

    If IDX doesn’t install it automatically via the provided .nix file, you can do it this way.

  2. Initialize a New Poetry Project
    Run the following command in your project’s root directory:

    poetry init

    Poetry will guide you through setting up your project.

  3. Add Packages with Poetry
    To add a package, use:

    poetry add pandas

    Poetry will manage everything and keep it all neat and tidy in a pyproject.toml file.

Why I Recommend Poetry

  • Dependency Tracking: Poetry locks down the exact versions of your packages, helping you avoid conflicts.
  • Reproducibility: It ensures anyone can recreate your environment exactly using the poetry.lock file.
  • Convenience: Poetry can manage your virtual environment and install packages automatically.

A .nix File to Automate IDX Setup

Here’s the .nix file I used to configure my IDX workspace. It ensures the right Python version, installs essential packages, and sets up useful extensions for data science workflows:

{ pkgs, ... }: {  
  channel = "stable-24.05";  
  packages = [  
    pkgs.python311  
    pkgs.python311Packages.pip  
  ];  
  idx = {  
    extensions = [  
      "ms-toolsai.jupyter"  
      "ms-python.python"  
      "charliermarsh.ruff"  
      "KevinRose.vsc-python-indent"  
      "ms-python.debugpy"  
      "tamasfe.even-better-toml"  
    ];  
    workspace = {  
      onCreate = ''  
        python -m venv .venv  
        source .venv/bin/activate  
        pip install poetry  
      '';  
    };  
  };  
}