Improve restarts

[canercidam]

• Creating and mounting specific Docker-managed volumes for container data instead of bind mounting
• Mounting validator logs to a Docker-managed volume so that we don't have removal issues on restart
• Identifying Docker Compose projects by the ID in manifest at start
• Taking down Docker Compose projects by the ID with automatic container and volume removal (and getting rid of removal based on the container list)
• errgroup use for starting services on the host

Fixes #181

[ferranbt]

I would prefer not to do this specially since we need this function anyway to clean all the containers related to playground if we have a playground clean command. Also, I try to avoid as much as possible having to rely on exec operations since they are harder to debug and reason about.

[canercidam]

I'm not a fan of doing everything with commands either 🙂 But we don't seem to have much choice for docker compose. If it's okay to start docker compose from a command, why not take it down from another (quicker) command? By the way, this cleans up only the specific session and not the other playground sessions in parallel and it leaves no containers, volumes, networks - it's super clean.

If we want to hook this up to playground clean somehow and use in detached mode, I think this command likely helps us - would you disagree?

[ferranbt]

My only concern is side effects of having to parse logs if we want to provide useful information on what it is happening on the output of the command while now it is pretty explicit if something fails what is going on.

I agree that this becomes helpful if we have to remove other artifacts from the execution (i.e. volumes, network) but I do not think that is requirement right now (see my other comment on using volumes).

At the same time, I am not sure how this can change be helpful for playground clean since it would need a reference to every docker-compose file? While, the current code does work for playground clean since you only have to modify the filter.

But, my concern is not major though (unlike using volumes). But would like to get your thoughts on this.

[canercidam]

It's not necessary to reference the compose file - just the session ID is sufficient because we are using that as the Docker Compose project name with the -p flag. And we are able to take stuff down by just providing -p <session-id> -v.

BTW, I think the GUIDs seem a little unfriendly for this purpose. We could probably use a truncated hex hash. WDYT?

[canercidam]

About the logs of this command: It is just a stop command so I expect any failure output to be minimal. And when it's all Docker Compose responsibility to turn things on/off, there is very little room for failure in my experience. I think it's useful to make builder-playground spit out any failure from this command as it is. The user can either try to reason about it alone (if savvy about Docker) or copy to us over an issue here.

[ferranbt]

I do not think is a good idea to add Docker volumes. It has been really helpful in the past to have everything self contained in the output folder:

• You can take snapshots of the recipe execution by zipping the output folder.
• You can debug easier what are the artifacts you are generating by inspecting the output folder contents.

I feel that if we introduce volumes we lose those capabilities.

[canercidam]

The cleanup on restart problem happens because one of the containers is writing a dir (and a file under that) to the artifacts dir with root permissions. I hear you about the use cases but, outside of the cleanup issue, we seem to risk mixing up container outputs from this dir as well. If we want to put artifacts into containers, we can also add that as another layer by using the Go Docker client that we already use - we don't have to do bind-mount to a single dir from all containers.

The alternatives are asking all Linux users to run sudo rm all the time or set up rootless Docker which makes the tool less friendly for a bunch of us. And using volumes like this is a step forward for all users to have better cleanups.

If somebody wants to see what artifacts the recipe generated, container inspection or docker export is already there for us. And the Docker volumes are not black boxes - they already always available from /var/lib/docker/volumes (we can prevent auto-removal of volumes with a flag). The user can either cd there to directly start browsing the files or we can provide a new command that spits out the right command to cd like $(builder-playground files <session-id>) or similar.