Project

General

Profile

Actions

Bug #935

open

[BE] Lỗi out of memory của GPU

Added by Lê An 3 months ago. Updated 2 months ago.

Status:
Resolved - Dev
Priority:
Normal
Assignee:
Category:
-
Start date:
07/04/2024
Due date:
% Done:

0%

Estimated time:
Spent time:

Description

Bối cảnh khi giao thoa file có sample audio file quá lớn >= 100MB thì hệ thống báo lỗi
Sample

{
    "trace_id": "149e62c6-81b8-4ea1-b630-7164d7e79166",
    "training_type": "open_voice",
    "type": "USER_VOICE",
    "speaker": 398,
    "input_audio_path": "flare_uploaded/2377/20240704_042550_744602/7e93b014-39bd-11ef-a4cb-baa161917d76.mp3",
    "output_audio_path": "flare_uploaded/2377/20240704_042550_744602/7e93b014-39bd-11ef-a4cb-baa161917d76_inferred.mp3",
    "callback_endpoint": "https://api-dev.ttsopenai.com/api/v1/voice-library/webhook/inference/149e62c6-81b8-4ea1-b630-7164d7e79166",
    "embedding_path": "voices_embedding/models/openvoice_UV29DD59B13E_embedding.pth",
    "sample_audio_path": "flare_uploaded/29/20240704_042433_745234/Oswald.mp3" 
}

Lỗi

Traceback (most recent call last):
  File "/app/app/services/open_voice_inference_service.py", line 36, in process
    self._prepare_embedding_file()
  File "/app/app/services/open_voice_inference_service.py", line 95, in _prepare_embedding_file
    self.open_voice_processor.generate_embedding(input_dir=generate_input_path, output_dir=generate_output_path)
  File "/app/app/services/voice_processor/models/openvoice.py", line 106, in generate_embedding
    embedd, _ = self.get_speaker_embedd(os.path.join(input_dir, filename))
  File "/app/app/services/voice_processor/models/openvoice.py", line 89, in get_speaker_embedd
    return se_extractor.get_se(
  File "/app/app/services/voice_processor/openvoice/se_extractor.py", line 156, in get_se
    return vc_model.extract_se(audio_segs, se_save_path=se_path), audio_name
  File "/app/app/services/voice_processor/openvoice/api.py", line 131, in extract_se
    g = self.model.ref_enc(y.transpose(1, 2)).unsqueeze(-1)
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/app/app/services/voice_processor/openvoice/models.py", line 348, in forward
    out = F.relu(out)  # [N, 128, Ty//2^K, n_mels//2^K]
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/functional.py", line 1473, in relu
    result = torch.relu(input)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 8.43 GiB. GPU 0 has a total capacity of 15.72 GiB of which 4.50 GiB is free. Process 479241 has 10.88 GiB memory in use. Process 479245 has 344.00 MiB memory in use. Of the allocated memory 9.61 GiB is allocated by PyTorch, and 1.07 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

Actions #1

Updated by Lê An 3 months ago

  • Status changed from New to In Progress
Actions #2

Updated by Lê An 2 months ago

  • Status changed from In Progress to Resolved - Dev
Actions

Also available in: Atom PDF