Project

General

Profile

Bug #593 » ec515b48-e43f-11ee-83e1-fe3e40614918.log

worker log - Trịnh Thế Định, 03/17/2024 11:10 AM

 
INFO:__main__:Worker started.
INFO:__main__:Parameters: Namespace(history_uuid='ec515b48-e43f-11ee-83e1-fe3e40614918')
INFO:__main__:conversion_history
INFO:__main__:<app.tts_common.models.history.ConversionHistory object at 0x7f3530302890>
INFO:__main__:current_user from db Quang Linh Đỗ Hoàng
ERROR:__main__:processing tts_history: ec515b48-e43f-11ee-83e1-fe3e40614918
ERROR:__main__:current user_id: 511
INFO:__main__:Try to download and decrypt if needed for ec515b48-e43f-11ee-83e1-fe3e40614918
INFO:__main__:File downloaded successfully to: tmp/contabo_uploaded/511/20240317_092245_362388/kiem lai 3.docx
INFO:Job: convert_and_send_result ec515b48-e43f-11ee-83e1-fe3e40614918:Convert TTS and send result START...
INFO:Job: convert_and_send_result ec515b48-e43f-11ee-83e1-fe3e40614918:conversion_history_uuid ec515b48-e43f-11ee-83e1-fe3e40614918
INFO:Job: convert_and_send_result ec515b48-e43f-11ee-83e1-fe3e40614918:input_file_path tmp/contabo_uploaded/511/20240317_092245_362388/kiem lai 3.docx
INFO:Job: convert_and_send_result ec515b48-e43f-11ee-83e1-fe3e40614918:model_price_rate 2
INFO:Job: convert_and_send_result ec515b48-e43f-11ee-83e1-fe3e40614918:file_extension docx
INFO:Job: convert_and_send_result ec515b48-e43f-11ee-83e1-fe3e40614918:current estimated_total_credits to calculate lock: 43812
INFO:Job: convert_and_send_result ec515b48-e43f-11ee-83e1-fe3e40614918:current_estimate_credits: 43812
0-kiem lai 3 tmp/contabo_uploaded/511/20240317_092245_362388/0-kiem lai 3.mp3
1-kiem lai 3 tmp/contabo_uploaded/511/20240317_092245_362388/1-kiem lai 3.mp3
INFO:app.tts_common.services.speech:Parent process: Process-2 START
INFO:app.tts_common.services.speech:setup_api_key of model: gpt-3.5
INFO:app.tts_common.services.speech:Parent process: Process-3 START
INFO:app.tts_common.services.speech:setup_api_key of model: gpt-3.5
INFO:app.tts_common.services.speech:count_all_active_key: 1260
INFO:app.tts_common.services.speech:Requesting openAI with key: ********************3tZFG2Zo9W4hjfK5Nm2EsUv2X
INFO:app.tts_common.services.speech:count_all_active_key: 1260
INFO:app.tts_common.services.speech:Requesting openAI with key: ********************kjinmzdn8u2dHBPF5M4Y89wjd
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/audio/speech "HTTP/1.1 400 Bad Request"
ERROR:app.tts_common.services.speech:Another non-200-range status code was received
ERROR:app.tts_common.services.speech:400
ERROR:app.tts_common.services.speech:<Response [400 Bad Request]>
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/audio/speech "HTTP/1.1 400 Bad Request"
ERROR:app.tts_common.services.speech:Another non-200-range status code was received
ERROR:app.tts_common.services.speech:400
ERROR:app.tts_common.services.speech:<Response [400 Bad Request]>
INFO:app.tts_common.services.speech:Requesting openAI with key: ********************3tZFG2Zo9W4hjfK5Nm2EsUv2X
INFO:app.tts_common.services.speech:Requesting openAI with key: ********************kjinmzdn8u2dHBPF5M4Y89wjd
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/audio/speech "HTTP/1.1 400 Bad Request"
ERROR:app.tts_common.services.speech:Another non-200-range status code was received
ERROR:app.tts_common.services.speech:400
ERROR:app.tts_common.services.speech:<Response [400 Bad Request]>
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/audio/speech "HTTP/1.1 400 Bad Request"
ERROR:app.tts_common.services.speech:Another non-200-range status code was received
ERROR:app.tts_common.services.speech:400
ERROR:app.tts_common.services.speech:<Response [400 Bad Request]>
INFO:app.tts_common.services.speech:Requesting openAI with key: ********************3tZFG2Zo9W4hjfK5Nm2EsUv2X
INFO:app.tts_common.services.speech:Requesting openAI with key: ********************kjinmzdn8u2dHBPF5M4Y89wjd
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/audio/speech "HTTP/1.1 400 Bad Request"
ERROR:app.tts_common.services.speech:Another non-200-range status code was received
ERROR:app.tts_common.services.speech:400
ERROR:app.tts_common.services.speech:<Response [400 Bad Request]>
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/audio/speech "HTTP/1.1 400 Bad Request"
ERROR:app.tts_common.services.speech:Another non-200-range status code was received
ERROR:app.tts_common.services.speech:400
ERROR:app.tts_common.services.speech:<Response [400 Bad Request]>
INFO:app.tts_common.services.speech:Requesting openAI with key: ********************3tZFG2Zo9W4hjfK5Nm2EsUv2X
INFO:app.tts_common.services.speech:Requesting openAI with key: ********************kjinmzdn8u2dHBPF5M4Y89wjd
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/audio/speech "HTTP/1.1 429 Too Many Requests"
ERROR:app.tts_common.services.speech:A 429 status code was received; we should back off a bit.
INFO:app.tts_common.crud.crud_openai_api_key:set_usable_at: sess-Xh11k6a2npNuwVJ3tZFG2Zo9W4hjfK5Nm2EsUv2X - gpt-3.5 - 2024-03-17 09:25:15.736456
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/audio/speech "HTTP/1.1 429 Too Many Requests"
ERROR:app.tts_common.services.speech:A 429 status code was received; we should back off a bit.
INFO:app.tts_common.crud.crud_openai_api_key:set_usable_at: sess-iGef6UotVblspIvkjinmzdn8u2dHBPF5M4Y89wjd - gpt-3.5 - 2024-03-17 09:25:15.853437
INFO:app.tts_common.services.speech:Requesting openAI with key: ********************kxSDIAsEqWiuLY565AbP6hBOl
INFO:app.tts_common.services.speech:Requesting openAI with key: ********************FvVLEMI339uiJvPKKb7HTQJmc
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/audio/speech "HTTP/1.1 400 Bad Request"
ERROR:app.tts_common.services.speech:Another non-200-range status code was received
ERROR:app.tts_common.services.speech:400
ERROR:app.tts_common.services.speech:<Response [400 Bad Request]>
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/audio/speech "HTTP/1.1 400 Bad Request"
ERROR:app.tts_common.services.speech:Another non-200-range status code was received
ERROR:app.tts_common.services.speech:400
ERROR:app.tts_common.services.speech:<Response [400 Bad Request]>
INFO:app.tts_common.services.speech:Requesting openAI with key: ********************kxSDIAsEqWiuLY565AbP6hBOl
INFO:app.tts_common.services.speech:Requesting openAI with key: ********************FvVLEMI339uiJvPKKb7HTQJmc
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/audio/speech "HTTP/1.1 400 Bad Request"
ERROR:app.tts_common.services.speech:Another non-200-range status code was received
ERROR:app.tts_common.services.speech:400
ERROR:app.tts_common.services.speech:<Response [400 Bad Request]>
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/audio/speech "HTTP/1.1 400 Bad Request"
ERROR:app.tts_common.services.speech:Another non-200-range status code was received
ERROR:app.tts_common.services.speech:400
ERROR:app.tts_common.services.speech:<Response [400 Bad Request]>
INFO:app.tts_common.services.speech:Requesting openAI with key: ********************kxSDIAsEqWiuLY565AbP6hBOl
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/audio/speech "HTTP/1.1 400 Bad Request"
ERROR:app.tts_common.services.speech:Another non-200-range status code was received
ERROR:app.tts_common.services.speech:400
ERROR:app.tts_common.services.speech:<Response [400 Bad Request]>
INFO:app.tts_common.services.speech:Requesting openAI with key: ********************FvVLEMI339uiJvPKKb7HTQJmc
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/audio/speech "HTTP/1.1 400 Bad Request"
ERROR:app.tts_common.services.speech:Another non-200-range status code was received
ERROR:app.tts_common.services.speech:400
ERROR:app.tts_common.services.speech:<Response [400 Bad Request]>
INFO:app.tts_common.services.speech:Requesting openAI with key: ********************kxSDIAsEqWiuLY565AbP6hBOl
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/audio/speech "HTTP/1.1 429 Too Many Requests"
ERROR:app.tts_common.services.speech:A 429 status code was received; we should back off a bit.
INFO:app.tts_common.crud.crud_openai_api_key:set_usable_at: sess-X0GlFyvfovJXCDOkxSDIAsEqWiuLY565AbP6hBOl - gpt-3.5 - 2024-03-17 09:25:28.563004
INFO:app.tts_common.services.speech:Requesting openAI with key: ********************FvVLEMI339uiJvPKKb7HTQJmc
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/audio/speech "HTTP/1.1 429 Too Many Requests"
ERROR:app.tts_common.services.speech:A 429 status code was received; we should back off a bit.
INFO:app.tts_common.crud.crud_openai_api_key:set_usable_at: sess-DtSbpidkBMSaQCCFvVLEMI339uiJvPKKb7HTQJmc - gpt-3.5 - 2024-03-17 09:25:28.804010
INFO:app.tts_common.services.speech:Requesting openAI with key: ********************6dey6Be5ceY7U1j6GQLaxcCUt
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/audio/speech "HTTP/1.1 400 Bad Request"
ERROR:app.tts_common.services.speech:Another non-200-range status code was received
ERROR:app.tts_common.services.speech:400
ERROR:app.tts_common.services.speech:<Response [400 Bad Request]>
INFO:app.tts_common.services.speech:Requesting openAI with key: ********************U34fEU7VstOTAU13FLetEiakj
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/audio/speech "HTTP/1.1 400 Bad Request"
ERROR:app.tts_common.services.speech:Another non-200-range status code was received
ERROR:app.tts_common.services.speech:400
ERROR:app.tts_common.services.speech:<Response [400 Bad Request]>
INFO:app.tts_common.helpers.translator_helper:get_boss_key response
INFO:app.tts_common.helpers.translator_helper:<Response [200]>
INFO:app.tts_common.helpers.translator_helper:https://iv0fyyh85j.execute-api.eu-central-1.amazonaws.com/prod/oakey/boss/f6e5740d-099c-4084-a454-bb339b407128
INFO:app.tts_common.helpers.translator_helper:{'success': True, 'api_key': 'sk-xzXro7PxD2MpQzw44b6329C566F24e129cBeBb05A99fDb1c', 'partitionKey': 'bossKey', 'created_at': '2023-08-01 11:02:34', 'id': 2, 'base_url': 'https://aihubmix.com/v1'}
INFO:app.tts_common.services.speech:Using boss key in request_text_openai_by_piece
INFO:app.tts_common.services.speech:Requesting openAI with key: ********************b6329C566F24e129cBeBb05A99fDb1c
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/audio/speech "HTTP/1.1 401 Unauthorized"
ERROR:app.tts_common.services.speech:The API key is invalid
ERROR:app.tts_common.services.speech:Error code: 401 - {'error': {'message': 'Incorrect API key provided: sk-xzXro***************************************Db1c. You can find your API key at https://platform.openai.com/account/api-keys.', 'type': 'invalid_request_error', 'param': None, 'code': 'invalid_api_key'}}
INFO:app.tts_common.crud.crud_openai_api_key:deactivate_api_key: sk-xzXro7PxD2MpQzw44b6329C566F24e129cBeBb05A99fDb1c - gpt-3.5
INFO:app.tts_common.helpers.translator_helper:get_boss_key response
INFO:app.tts_common.helpers.translator_helper:<Response [200]>
INFO:app.tts_common.helpers.translator_helper:https://iv0fyyh85j.execute-api.eu-central-1.amazonaws.com/prod/oakey/boss/f6e5740d-099c-4084-a454-bb339b407128
INFO:app.tts_common.helpers.translator_helper:{'success': True, 'api_key': 'sk-xzXro7PxD2MpQzw44b6329C566F24e129cBeBb05A99fDb1c', 'partitionKey': 'bossKey', 'created_at': '2023-08-01 11:02:34', 'id': 2, 'base_url': 'https://aihubmix.com/v1'}
INFO:app.tts_common.services.speech:Using boss key in request_text_openai_by_piece
INFO:app.tts_common.services.speech:Requesting openAI with key: ********************b6329C566F24e129cBeBb05A99fDb1c
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/audio/speech "HTTP/1.1 401 Unauthorized"
ERROR:app.tts_common.services.speech:The API key is invalid
ERROR:app.tts_common.services.speech:Error code: 401 - {'error': {'message': 'Incorrect API key provided: sk-xzXro***************************************Db1c. You can find your API key at https://platform.openai.com/account/api-keys.', 'type': 'invalid_request_error', 'param': None, 'code': 'invalid_api_key'}}
INFO:app.tts_common.crud.crud_openai_api_key:deactivate_api_key: sk-xzXro7PxD2MpQzw44b6329C566F24e129cBeBb05A99fDb1c - gpt-3.5
INFO:app.tts_common.helpers.tts_history_helper:add_percent_complete: ec515b48-e43f-11ee-83e1-fe3e40614918 - 50%
INFO:[CRUD History]:add_percent_complete: ec515b48-e43f-11ee-83e1-fe3e40614918 - 50%
INFO:app.tts_common.services.speech:Parent process: Process-2 END
complete_count 0
total_pieces 2
INFO:app.tts_common.helpers.tts_history_helper:add_percent_complete: ec515b48-e43f-11ee-83e1-fe3e40614918 - 100%
INFO:[CRUD History]:add_percent_complete: ec515b48-e43f-11ee-83e1-fe3e40614918 - 100%
INFO:app.tts_common.services.speech:Parent process: Process-3 END
complete_count 1
total_pieces 2
ERROR:__main__:Convert and send result failed
ERROR:__main__:[Errno 2] No such file or directory: 'tmp/contabo_uploaded/511/20240317_092245_362388/0-kiem lai 3.mp3'
Traceback (most recent call last):
File "/app/worker.py", line 153, in main
convert_result = await convert_and_send_result(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/app/tts_common/jobs/convert_and_send_result.py", line 151, in convert_and_send_result
audio = AudioSegment.from_mp3(file)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pydub/audio_segment.py", line 796, in from_mp3
return cls.from_file(file, 'mp3', parameters=parameters)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pydub/audio_segment.py", line 651, in from_file
file, close_file = _fd_or_path_or_tempfile(file, 'rb', tempfile=False)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pydub/utils.py", line 60, in _fd_or_path_or_tempfile
fd = open(fd, mode=mode)
^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'tmp/contabo_uploaded/511/20240317_092245_362388/0-kiem lai 3.mp3'
ERROR:__main__:Traceback (most recent call last):
File "/app/worker.py", line 153, in main
convert_result = await convert_and_send_result(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/app/tts_common/jobs/convert_and_send_result.py", line 151, in convert_and_send_result
audio = AudioSegment.from_mp3(file)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pydub/audio_segment.py", line 796, in from_mp3
return cls.from_file(file, 'mp3', parameters=parameters)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pydub/audio_segment.py", line 651, in from_file
file, close_file = _fd_or_path_or_tempfile(file, 'rb', tempfile=False)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pydub/utils.py", line 60, in _fd_or_path_or_tempfile
fd = open(fd, mode=mode)
^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'tmp/contabo_uploaded/511/20240317_092245_362388/0-kiem lai 3.mp3'
Traceback (most recent call last):
File "/app/worker.py", line 153, in main
convert_result = await convert_and_send_result(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/app/tts_common/jobs/convert_and_send_result.py", line 151, in convert_and_send_result
audio = AudioSegment.from_mp3(file)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pydub/audio_segment.py", line 796, in from_mp3
return cls.from_file(file, 'mp3', parameters=parameters)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pydub/audio_segment.py", line 651, in from_file
file, close_file = _fd_or_path_or_tempfile(file, 'rb', tempfile=False)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pydub/utils.py", line 60, in _fd_or_path_or_tempfile
fd = open(fd, mode=mode)
^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'tmp/contabo_uploaded/511/20240317_092245_362388/0-kiem lai 3.mp3'
ERROR:__main__:ec515b48-e43f-11ee-83e1-fe3e40614918
Traceback (most recent call last):
File "/app/worker.py", line 153, in main
convert_result = await convert_and_send_result(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/app/tts_common/jobs/convert_and_send_result.py", line 151, in convert_and_send_result
audio = AudioSegment.from_mp3(file)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pydub/audio_segment.py", line 796, in from_mp3
return cls.from_file(file, 'mp3', parameters=parameters)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pydub/audio_segment.py", line 651, in from_file
file, close_file = _fd_or_path_or_tempfile(file, 'rb', tempfile=False)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pydub/utils.py", line 60, in _fd_or_path_or_tempfile
fd = open(fd, mode=mode)
^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'tmp/contabo_uploaded/511/20240317_092245_362388/0-kiem lai 3.mp3'
INFO:app.tts_common.helpers.email_helper:Send translated file failed notice email
INFO:app.tts_common.helpers.email_helper:sent to message to kafka topic: tts-emails-prod
INFO:Firebase client:Add notification to firebase START
INFO:Firebase client:Add notification to firebase END
INFO:__main__:Recalculate user locked credit
INFO:__main__:Convert and send result END
results save text pieces to files prepare to join audios
Produced event to topic tts-emails-prod: value = {"function": "send_translated_file_failed_notice", "email_to": "dolinhluat@gmail.com", "recipient_name": "Quang Linh \u0110\u1ed7 Ho\u00e0ng", "file_name": "kiem lai 3.docx"}
(3-3/6)