Human versus Machine Performance in Text Reading and Understanding

Human versus Machine Performance in Text Reading and Understanding

At YUKKA Lab we are often asked how the YUKKA analysis of documents compares to a human analyst reading the same documents. In this blog article, we address this question by comparing human and machine performance in text reading and understanding.

How do humans read?

When reading, human eyes fixate at certain positions in the text moving swiftly further. Knowledge of the language allows performing structural analysis of the text almost simultaneously, leading to a representation of the meaning. World and domain knowledge provide the necessary context both for producing the correct analysis (e.g. in disambiguation when several meanings are possible) and making full sense of the message. The message sometimes cannot be fully interpreted without taking into consideration the preceding discourse. Attention and short-term memory play an important role when connecting the new content to the already processed one, e.g. by resolving the pronoun “he” as referring to a particular person. Through common-sense reasoning, the reader also infers information that has not been explicitly conveyed by the author. The interpreted content is then stored as a set of facts in long-term memory and after a while, a human does not remember the exact details of the text anymore, but rather the gist. After an even more extended period, only a subset of the facts learned when reading the text remains as part of the world/domain knowledge of the reader, who may not even remember anymore the source where from he read the information.

How do machines read?

A typical NLP system processes text sequentially. It performs a linguistic analysis of the text at different levels of increasing complexity. At the lowest level, it tokenizes the text into words, it splits the text into sentences, it performs lemmatization, morphological analysis and part-of-speech tagging of the words. This low-level information is used to obtain a syntactic-semantic representation of the sentences (a representation of their meaning). A knowledge-base, such as an ontology, allows connecting the text to the universe of entities we are interested in. With the knowledge-base, named entity recognition (NER) in the text can be performed quite precisely and the named entities can be linked to real-world entities. Many entities in the text, however, are mentioned only implicitly, e.g. through a pronoun (“he”) or a generic expression (“the company”). Through the co-reference resolution, the NLP system is able to establish the connection between the implicit expression and the entity it refers to. On the basis of this representation of entities and the relations holding between them provided by the semantic analysis, an application-dependent higher level of analysis can be performed, such as the sentiment and target recognition carried out by the YUKKA LAB system, in which the opinions expressed in the text, the polarity of those opinions (positive, negative or neutral) and the target of the opinion (the entity, about which the opinion holds) are detected.

When comparing human and computer reading performance, the most important advantage of computers is, of course, the speed. A human reads 200-400 words per minute. The YUKKA system, for example, with a cluster of several servers, is able to process 38.000 words per minute. With an average financial document length of 450 words, this amounts to 83 documents per minute and 120.000 documents per day. In a period of two weeks, the YUKKA Opinion Detection engine can read and understand 140 million documents. A human could never possibly read such an amount of documents, not even in a lifetime.

A Human forgets

Computers are also the absolute winners when it comes to the retention rate. For the machine, there is no memory limit for storing the results of the document reading and even the details of the analysis as well. Those results can be accessed at any time later, either in an aggregated way (e.g., with charts and statistics or enriched with more information) or on a single document basis. A human, however, as we have seen, only retains part of the knowledge acquired through reading.

A further reason for possibly preferring computers over humans on reading tasks is that, while humans interpret texts from their perspective and state of mind, computers can provide a rational assessment. However, note that computer assessments can also be biased by the data used to train their models.

Computers may outperform humans in a particular area of knowledge. For example, the YUKKA LAB engine currently has a large, and continuously growing, knowledge base of entities (companies, persons, organizations, …) relevant in the financial domain and facts related to them. The YUKKA system may already have more factual information than a human expert in the financial domain.

About the disadvantages machines are facing

But not all are advantages on the machine side. When it comes to adapting to new domains, computers are very limited because real-world knowledge acquisition remains a problem. Humans, on the other hand, have a broad understanding of the world and can quickly adapt to new domains.

Humans can also read between the lines, that is, they can grasp information which has not been stated explicitly, while computers still can do this only to a limited extent and for specific domains, i.e., computers have difficulties understanding misspellings and irony.

Finally, computers cannot easily distinguish the relevant parts of the text from the details, and therefore, text summarization remains a challenge for them, while humans can easily extract relevant information and reproduce the gist of the text with their own words.

Dr. Núria Bertomeu Castelló wrote this story. She is Director of Natural Language Processing at YUKKA Lab and did her Ph.D. in computational linguistics at Saarland University.



Human versus Machine Performance in Text Reading and Understanding

Human versus Machine Performance in Text Reading and Understanding

At YUKKA Lab we are often asked how the YUKKA analysis of documents compares to a human analyst reading the same documents. In this blog article, we address this question by comparing human and machine performance in text reading and understanding.

How do humans read?

When reading, human eyes fixate at certain positions in the text moving swiftly further. Knowledge of the language allows performing structural analysis of the text almost simultaneously, leading to a representation of the meaning. World and domain knowledge provide the necessary context both for producing the correct analysis (e.g. in disambiguation when several meanings are possible) and making full sense of the message. The message sometimes cannot be fully interpreted without taking into consideration the preceding discourse. Attention and short-term memory play an important role when connecting the new content to the already processed one, e.g. by resolving the pronoun “he” as referring to a particular person. Through common-sense reasoning, the reader also infers information that has not been explicitly conveyed by the author. The interpreted content is then stored as a set of facts in long-term memory and after a while, a human does not remember the exact details of the text anymore, but rather the gist. After an even more extended period, only a subset of the facts learned when reading the text remains as part of the world/domain knowledge of the reader, who may not even remember anymore the source where from he read the information.

How do machines read?

A typical NLP system processes text sequentially. It performs a linguistic analysis of the text at different levels of increasing complexity. At the lowest level, it tokenizes the text into words, it splits the text into sentences, it performs lemmatization, morphological analysis and part-of-speech tagging of the words. This low-level information is used to obtain a syntactic-semantic representation of the sentences (a representation of their meaning). A knowledge-base, such as an ontology, allows connecting the text to the universe of entities we are interested in. With the knowledge-base, named entity recognition (NER) in the text can be performed quite precisely and the named entities can be linked to real-world entities. Many entities in the text, however, are mentioned only implicitly, e.g. through a pronoun (“he”) or a generic expression (“the company”). Through the co-reference resolution, the NLP system is able to establish the connection between the implicit expression and the entity it refers to. On the basis of this representation of entities and the relations holding between them provided by the semantic analysis, an application-dependent higher level of analysis can be performed, such as the sentiment and target recognition carried out by the YUKKA LAB system, in which the opinions expressed in the text, the polarity of those opinions (positive, negative or neutral) and the target of the opinion (the entity, about which the opinion holds) are detected.

When comparing human and computer reading performance, the most important advantage of computers is, of course, the speed. A human reads 200-400 words per minute. The YUKKA system, for example, with a cluster of several servers, is able to process 38.000 words per minute. With an average financial document length of 450 words, this amounts to 83 documents per minute and 120.000 documents per day. In a period of two weeks, the YUKKA Opinion Detection engine can read and understand 140 million documents. A human could never possibly read such an amount of documents, not even in a lifetime.

A Human forgets

Computers are also the absolute winners when it comes to the retention rate. For the machine, there is no memory limit for storing the results of the document reading and even the details of the analysis as well. Those results can be accessed at any time later, either in an aggregated way (e.g., with charts and statistics or enriched with more information) or on a single document basis. A human, however, as we have seen, only retains part of the knowledge acquired through reading.

A further reason for possibly preferring computers over humans on reading tasks is that, while humans interpret texts from their perspective and state of mind, computers can provide a rational assessment. However, note that computer assessments can also be biased by the data used to train their models.

Computers may outperform humans in a particular area of knowledge. For example, the YUKKA LAB engine currently has a large, and continuously growing, knowledge base of entities (companies, persons, organizations, …) relevant in the financial domain and facts related to them. The YUKKA system may already have more factual information than a human expert in the financial domain.

About the disadvantages machines are facing

But not all are advantages on the machine side. When it comes to adapting to new domains, computers are very limited because real-world knowledge acquisition remains a problem. Humans, on the other hand, have a broad understanding of the world and can quickly adapt to new domains.

Humans can also read between the lines, that is, they can grasp information which has not been stated explicitly, while computers still can do this only to a limited extent and for specific domains, i.e., computers have difficulties understanding misspellings and irony.

Finally, computers cannot easily distinguish the relevant parts of the text from the details, and therefore, text summarization remains a challenge for them, while humans can easily extract relevant information and reproduce the gist of the text with their own words.

Dr. Núria Bertomeu Castelló wrote this story. She is Director of Natural Language Processing at YUKKA Lab and did her Ph.D. in computational linguistics at Saarland University.