Text summarization with neural networks

Text Summarization

It is difficult for a person to manually select a summary in a large volume of text. Therefore, NLP raises the problem of creating an accurate and concise summary for the original document. Text Summarization is an important tool to help interpret textual information. Push notifications and article digests are getting a lot of attention, and the number of tasks to create smart and accurate summaries for large snippets of text is growing day by day.

Automatic extraction of summary from text works as follows. First, the frequency of occurrence of a word in a full text document is considered, then the 100 most frequent words are saved and sorted. After that, each sentence is evaluated according to the number of frequently used words, and the weight of the more frequently occurring word is higher. Finally, the first X sentences are sorted according to their position in the original text.

NLP

While maintaining its simplicity and generalizing ability, the automatic summary extraction algorithm is able to work in difficult situations. For example, many implementations fail on texts with foreign languages ​​or unique dictionary associations that are not contained in standard text arrays.

There are two fundamental approaches to shortening text: extractable and abstract. The first one extracts words and phrases from the original text to create a resume. The latter studies the internal linguistic representation in order to create a human-like presentation by paraphrasing the original text.