Technical

Does the docker container launch with --privileged flag?

No, the Docker container does not launch with the --privileged flag

Why do we mount the Docker socket while starting the containers?

The platform manages the device states and usage through the orchestration of docker containers. We require to mount the docker socket for managing docker containers on the worker node. This is mandatory for the platform and there are currently no plans or alternatives to remove this

Isn't mounting docker socket and --privileged flag the same?

while the --privileged flag gives broad system access to a container, mounting the Docker socket gives the container control over Docker on the host

Why do you use docker containers?

Our platform enables clustering gpu compute and provides the end user of the platform with a production ready environment to do distributed training. The custom docker images contain all the required drivers and environment with all libraries installed which enables efficient utilization of GPU and CPU resources which are mandatory for distributed training. Reproduction of the same environment as ours from a worker stand point is very troublesome and can have irregularities based on the platform the worker has (linux, windows), the most important requirement for distributed training is to exactly replicate the environment on all nodes.