Manta microservices should restart automatically on failure.

Current State:

1. When a Manta user scans a large amount of SAS scripts or MS SQL tables, Manta Flow Server crashes due to out of memory (TS016760059). The cause of the memory issue is being addressed by the issue, but this product idea is this: a non admin user is able to crash Manta Flow server, and it requires manual intervention from an admin to SSH into the Manta machine and restart Flow Server

2. Since Flow server crashed, and not Manta Launcher, the systemd unit script that starts MantaLauncher does not fail, and therefore systemd is unaware that Manta has crashed. Therefore systemd does not autostart Manta. Any OS level alerting systems also do not realise the crash.

Desired state:

1. If any manta microservice (except Manta Launcher) such as the Flow Server were to crash, Manta Launcher should automatically start up the failed service. It should try this a fixed number of time (eg: 5) and if Manta still doesnt start, it should print a log message, stop all of Manta and itself.

2. Alternatively, (if 1 is not feasible) customers should be given seperate systemd unit files to start each Manta microservice individually, so that if these services crash, systemd can start them automatically (the ability to restart is built into systemd, which is standard part of most Linux distributions)

This is very important from our perspective as we want to provide Manta as a platform at our enterprise, and the platform should be highly available. The auto starting of failed services is critical in making Manta reliable.

There are no good viable alternatives. When Manta crashes, it may take several hours for the admin to be alerted and restart the system, which erodes truist on the product.