Project

General

Profile

Actions
Copy link

Feature #1019

open

[infra] Voice-clone detect SQS connection lost

Added by Phước Ngọc Trần about 1 month ago. Updated 27 days ago.

Status:
In Progress
Priority:
Normal
Assignee:
Phước Ngọc Trần
Category:
-
Start date:
08/24/2024
Due date:
08/25/2024 (26 days late)
% Done:

50%

Estimated time:
3:00 h

Description

- GPU mất mạng trong khoảng thời gian dài thì sẽ bị mất connection. Cần có cơ chế detect connection bị mất ở livenessProbe Kubernetes. Chủ động check connection đến SQS. Nếu gặp error sẽ restart lại container.

Actions
Copy link
 #1

Updated by Phước Ngọc Trần 27 days ago

Viết 1 script python (healthcheck.py) để check connection.
LivenessProbe Kubernetes sẽ thực thi Script healthcheck.py 30s/1 lần để healthcheck.
Nếu health check fail => Restart container
Nếu health check success => Ko làm gì cả
https://github.com/AINNATE-TTS/voice_clone_api/pull/117

Actions
Copy link
 #2

Updated by Phước Ngọc Trần 27 days ago

  • Due date changed from 08/10/2024 to 08/25/2024
  • Status changed from New to In Progress
  • Start date changed from 08/10/2024 to 08/24/2024
  • % Done changed from 0 to 50
Actions
Copy link

Also available in: Atom PDF