Public-Mirror/LunaTranslator
1
0
Fork 0
mirror of https://github.com/HIllya51/LunaTranslator.git synced 2024-12-28 08:04:13 +08:00
Code Issues Projects Releases Wiki Activity

Add EOS token at end of sentence to prevent repetition. (#490)

Browse Source 
Fix repetition at the end of sentence by adding a <EOS> token id.
This commit is contained in:
chaihahaha 2024-01-12 22:43:43 +08:00 committed by GitHub
parent 734e6782d7
commit f06244e125
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 1 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
LunaTranslator/LunaTranslator/translator/ort_sp.py
View File

@ -158,6 +158,7 @@ class TS(basetrans):
) )
input_ids_len = n_tokens.value input_ids_len = n_tokens.value
input_ids_py = [token_ids[i] for i in range(input_ids_len)] input_ids_py = [token_ids[i] for i in range(input_ids_len)]
input_ids_py += [1] # add EOS token to notify the end of sentence and prevent repetition
return input_ids_py return input_ids_py
def decode_from_ids(self, output_ids_py): def decode_from_ids(self, output_ids_py):