degeneration

mentioned fine-grained control

train on seq of raw text prepended with control codes

famous for story generation

new loss

seq-level & token-level

new loss

new loss

random sample

beam search

heuristic random sample

heuristic beam search - beast first beam search

heuristic guided decoding strategy

raise diversity

fine-grained control including topic, end, repeatness

random sample+beam search

select based on conditional entropy

conbine similarity and beam search

intrinsic uncertainty could be solved by increase num_beam
harder to be trapped in local optimum

different task would have different optimal decoding strategy

high quality and high prob positive/negative correlation upon different task

appropriate information entropy is decisive upon quality (encourage model to generate tokens based on information)
information theory related

idea : gamma sampling treat equally every homoionyms
this new sample method treat homoionyms differently sorted by their cosine similarity towards keywords appeared in source text

change topic_tokens to key_words directly : works

add key_gamma : doesn't work

change probs before gamma sampling

delete the extreme condition dealing : works