Skip to content

Machine learning platform

The Machine Learning Platform (MLP) provides compute, storage and expertise to the machine learning and AI community in Switzerland, with the main user being the Swiss AI Initiative.

Getting started

Getting access

Project administrators (PIs and deputy PIs) of projects on the MLP can to invite users to join their project, before they can use the project's resources on Alps. This is performed using the project management tool

Once invited to a project, you will receive an email, which you can need to create an account and configure multi-factor authentication (MFA).

Systems

The main cluster provided by the MLP is Clariden, a large Grace-Hopper GPU system on Alps.

  • Bristen

    Bristen is a smaller system with A100 GPU nodes for data processing, development, x86 workloads and inference services.

File Systems and Storage

There are three main file systems mounted on the MLP clusters Clariden and Bristen.

type mount filesystem
Home /users/$USER VAST
Scratch /iopstor/scratch/cscs/$USER Iopstor
Project /capstor/store/cscs/swissai/<project> Capstor

Home

Every user has a home path ($HOME) mounted at /users/$USER on the VAST filesystem. The home directory has 50 GB of capacity, and is intended for configuration, small software packages and scripts.

Scratch

Scratch filesystems provide temporary storage for high-performance I/O for executing jobs. Use scratch to store datasets that will be accessed by jobs, and for job output. Scratch is per user - each user gets separate scratch path and quota.

  • The environment variable SCRATCH=/iopstor/scratch/cscs/$USER is set automatically when you log into the system, and can be used as a shortcut to access scratch.

scratch cleanup policy

Files that have not been accessed in 30 days are automatically deleted.

Scratch is not intended for permanant storage: transfer files back to the capstor project storage after job runs.

Note

There is an additional scratch path mounted on Capstor at /capstor/scratch/cscs/$USER, however this is not reccomended for ML workloads for performance reasons.

Project

Project storage is backed up, with no cleaning policy: it provides intermediate storage space for datasets, shared code or configuration scripts that need to be accessed from different vClusters. Project is per project - each project gets a project folder with project-specific quota.

  • if you need additional storage, ask your PI to contact the CSCS service managers Fawzi or Nicholas.
  • hard limits on capacity and inodes prevent users from writing to project if the quota is reached - you can check quota and available space by running the quota command on a login node or ela
  • it is not recommended to write directly to the project path from jobs.

Guides and tutorials

Todo

links to tutorials and guides for ML workflows