AI

GPT-3 and Conversational AI Models

zrik — Mon, 25 Jan 2021 07:20:49 +0000

GPT-3 and Conversational AI Models zrik Mon, 01/25/2021 - 10:20

Conversational AI is all about making machines communicate in as much human as it can with the understanding of natural language. They can be chatbots, voice bots, virtual assistants, and more. In reality, they may be slightly different from each other based on the language and data they trained on. However, one key feature that ties them all together is their ability to understand natural language and respond to it.

GPT-3 (Generative Pre-trained Transformer 3) is a language model made by OpenAI a research lab that’s funded by big tech companies like Microsoft and Google. It was released to the public through an API in July 2020. It’s based on a famous deep learning architecture called transformers published in 2015 and It's generative because unlike other neural networks that spit out a numeric score or a yes or no answer, it can generate long sequences of the original text as its output.

GPT-3 can do a lot of stuff like question answering, summarizing articles, information retrieval, it can also provide you with code snippets! What makes it unique It’s the size and the development it went through which made it the closest model to human performance.

How to make your GPT

To create our GPT language model we need to find out what do we need and how to do it. As we know, the main building blocks for deep learning are the dataset and computing power. So, let’s see some examples of that.

Starting with the dataset used to train GPT-3. OpenAI Has collected the data from the internet which generated about 499 billion tokens(word) comparing to GPT-2 which trained with 10 billion tokens which were about 40GB. That makes GPT-3 trained with a total of 2TB of data. Here is the breakdown of the data:

DataSet	# Tokens (Billions)
Total	499
Common Crawl (filtered by quality)	410
Web Text2	19
Books 1	12
Books 2	55
Wikipedia	3

Continuing with computing power, OpenAI found that to do well on their increasingly large datasets, they had to add more and more weights. The original Transformer from Google (BERT) had 110 million weights. GPT-1 followed this design. With GPT-2 It went up to 1.5 billion weights. With the latest GPT-3, the number of parameters has reached 175 billion, making GPT-3 the biggest neural network in the world.
Having a model with 175 billion weights is not a simple operation of increasing parameters but it becomes an incredible exercise in parallel computer processing. You can see how much it compares to others from figure 1.

Figure 1. GPT-3 training chart compared to others
Source from here

It hasn't described the exact computer configuration used for training, but others mentioned it was on a cluster of Nvidia V100 chips running in Microsoft Azure. The company described the total compute cycles required, stating that it is the equivalent of running one thousand trillion floating-point operations per second per day for 3,640 days. To make it simpler to imagine it. It would take about 3 years for the V100(highest GPU available) with an estimated cost of 4.6$ million for a single training run.

That’s not the only problem for the computing power but also the capacity to carry a 175 billion weight parameter each parameter is a float number of sizes 32bit which requires in total about 700GB of GPU Ram. 10 times more than the memory on a single GPU.

If you still up for the challenge and want to experiment with a GPT model. You can try GPT-Neo from here. it’s an implementation of model & data-parallel GPT2 & GPT3-like models, with the ability to scale up to full GPT3 sizes (and possibly more!), using the mesh-TensorFlow library.

But is it really worth it? We will discuss next the limitation of it.

Limitations of GPT-3

So what can GPT-3 do? Well, for one thing, it can answer questions on any topic while retaining the context of previous questions asked. We can see in figure 2 examples of questions GPT-3 got right.

Figure 2. GPT-3 answering correctly
Source from here

So, when does the GPT-3 model fails?

GPT-3 knows how to have a normal conversation. but It doesn’t quite know how to say “Wait a moment your question is nonsense.” It also doesn’t know how to say “I don’t know.” in figure 3

Figure 3. GPT-3 answering nonsense questions
Source from here

People are used to computers being super smart at logical activities, like playing chess or adding numbers. It might come as a surprise that GPT-3 is not perfect at simple math the questions from figure 4.

Figure 4. GPT-3 confused from simple math
Source from here

This problem shows up in more human questions as well, if you ask it about the result of a sequence of operations. You can see that in figure 5

Figure 5. GPT-3 fails on sequence of operations.
Source from here

It’s like GPT-3 has limited short-term memory, and has trouble reasoning about more than one or two objects in a sentence.

The biggest problem for GPT-3 is being a black box like most neural networks, it’s so captivating because it can answer such a vast array of questions right, but it also gets quite a bit of them wrong, as we previously saw. The problem is when GPT-3 falls short, there is no way of debugging it or pinpoint the source of the error. Any customer-facing interface which cannot be iterated and revised is unsustainable or scalable in a business environment.

This is another aspect in which the current conversational AI solutions are better to GPT-3. Even simple chatbots allow their users to alter and improve their conversational flows as needed. In the case of more sophisticated conversational AI interfaces, users not only have a clear snapshot of the error but can track down, diagnose, and remedy the issue instantly.

Conclusion

GPT-3 is quite impressive in some areas, and still clearly subhuman in others but it’s far from ready to be used on real products, and as for businesses looking to provide their audiences with engaging, timely, and helpful conversational experiences will continue to rely on existing conversational AI solutions.

I still hope that with a better understanding of its strengths and weaknesses, we data scientists will be better equipped to use modern language models like GPT-3 in a business environment with real products.

Resources:

Banner image

GPT-3.png

Introduction to Artificial Intelligence

zrik — Wed, 06 Jan 2021 14:56:23 +0000

Introduction to Artificial Intelligence zrik Wed, 01/06/2021 - 17:56

For sure you have heard the term “Artificial Intelligence” (AI) here or there, very repeatedly. They said it solves many problems, might take the human position in the future.
For what they are speaking about it day and night? for what reason we need it? what is it? How did it start?

Why do we need AI?

In this world we have lots of problems that we wish from the bottom of our hearts that it could be solved. Let’s concentrate on some of them here: Investigating the medical errors in USA shows that 5% of adults are misdiagnosed yearly. This misdiagnosis is the reason for 10% of patients’ deaths. Diagnosis is hard, experienced doctors are not always available and affordable, and will not forget the human errors. Moreover, doctors have their reasons to make errors with high pressure daily tasks.

Going to another disaster happening in the world, buildings collapsing. Please have a look on this chart that shows the number of deaths cause by structural collapses in India during the interval 2003-2012, it exceeds 2,000 persons. Based on the five reasons mentioned in this article, we can tell that the lack of experience, human errors, and cheating are the main reasons for building collapses around the world.

We all know about the high rate of deaths resulted by car accidents yearly, and very clearly 94% of car accidents are caused by human choices or errors.

Yes, the dream came true with AI. With AI we can get personalized-healthcare and person-centered medicine. AI will not only be capable of solving building collapses problem but also will make it possible that each one of us could afford one of Zaha Hadid designs. Self-driving cars usage in USA decreased the accidents count by 90%.

AI solves the problems and comes with extra features that we used to dream in. You will not be forced to waste your time again in shopping, AI will pick exactly the suit that you want for that specific ceremony. AI will also pick the missing food from refrigerator, based on your needs. AI will go beyond that, by reserving your wife's best resort at your anniversary. You do not need to be worry again about forgetting that date, AI will take care about you.
To sum up what AI can do for us:
• Make the most experienced human skills affordable by all human beings
• Saving our time, by performing the tedious work tailored by our needs and wishes

So, what is AI? and how it could solve our problems in this smart way?

What is AI?

We, humans, do like humans. Therefore, we worked on making computers think and learn like us. AI in terms, is the field that enables computers to solve human problems using the smartest human methods. The previous methods used to solve problems with computers was done by programming each case with the related required action. However, this is an impossible task with cases that has various probabilities, and sometimes hidden ones. Therefore, guiding the computers moved from feeding it with instructions to feeding it with examples. Then, the computer job will be to find the reasons for each scenario based on statistics and probabilities. And provide us with its conclusions and do the suitable action for that specific scenario.
When AI research started? by whom? and how did progress till now?

AI History

AI research started many decades ago. Alan Turing described AI literally in the same way it works nowadays in a public lecture in 1947: “What we want is a machine that can learn from experience” and that the “possibility of letting the machine alter its own instructions provides the mechanism for this.”.

Turing also proposed a test for identifying the expected intelligent machine if it has reached to the level that he suggested or not. This test is called Turing Test. There are currently some machines passed the Turing test. I expect that you have used one at least once. It is the chatbot. Chatbot model is a great example of AI machines that replies on human questions in a way that we will not know if it is a human or a machine reply. This was Turing test proposal.

Yes, you are right AI research has started many decades ago. Why only recently we started hearing about it, repeatedly? AI was growing during the past century very slowly and had stopped many times, (many AI winters). The two main reasons for proceeding slowly are the two lacks in: the high-performance machines and the data. In the last two decades with the rise of the interactive sites as social media, there is no lack in data anymore. And the high-performance machines became not only available but affordable to lots of scientists and startups across the world. The following infographic shows the ebbs and flows in AI history. More information about AI history can be found in “The history of Artificial Intelligence”.

AI Types

AI is categorized to two types, based on machine performance in various intelligent tasks, not in one specific task. As both types supposed to exceed human capabilities in one specific task at least.

1. Weak AI or Narrow AI (ANI):
It is the AIs that solves one specific task. It is needed for the time-consuming tasks and the tasks that we are incapable of solving because of the complex relations between the cause and result. We have many of weak AIs in these days, starting from predicting your next word when you write on your smartphone, and ending with the autonomous vehicles (self-driving cars).

2. Strong AI or Artificial General Intelligence (AGI):
It is still a theoretical AI type. As it is the form of human like intelligent machines. AGI mimics the human in terms of self-consciousness, reasoning, solving the problems, acting, and planning. If you want to imagine how AGI will perform in the future I, Robot movie might help.

We still do not know how AI learns from examples, this will be our next topic. It is Machine Learning. Stay tuned!

1.   Why is artificial intelligence important? https://www.sas.com/en_us/insights/analytics/what-is-artificial-intelligence.html
2.   Intro to AI: https://www2.slideshare.net/ankit_ppt/lesson-1-intro-to-ai-132216012?from_action=save
3.   AI History: http://sitn.hms.harvard.edu/flash/2017/history-artificial-intelligence/
4.   AI Types: https://www.ibm.com/cloud/learn/strong-ai#toc-strong-ai--YaLcx8oG
5.   Machine Learning