01 – A Preliminary Study on Generative AI and Large Language Models
Imagine being able to provide just a few keywords and having a computer write an article, design an image, or generate a piece of code for you. This is the capability of generative AI. It is not merely a cold tool; it is an intelligent partner that can understand your needs, unleash creativity, and respond accordingly. Forget about complex programming syntax—simply describe your ideas in everyday language, and generative AI can help turn them into reality.
In this section, we will use clear and accessible language to showcase the fascinating aspects of generative artificial intelligence (referred to as GenAI or AIGC) and discuss the new opportunities it presents in the field of programming.
How Do We Understand Generative AI?
The Evolution of Generative AI
The concept of generative AI dates back several decades, originating from scientists’ attempts to enable machines to simulate human communication. Early endeavors included the creation of rule-based chatbots, which quickly revealed their limitations. Over time, advancements in statistics and machine learning laid a solid foundation for the evolution of AI.
Notably, the emergence of machine learning in the 1990s marked a milestone in the history of AI development. Statistical models enabled machines to learn patterns from textual data without the need for hard coding. This advancement significantly improved machines’ efficiency in understanding human language and classifying input text.
The Rise of Neural Networks and Intelligent Assistants
With advancements in computing hardware, we gained the ability to process larger datasets and execute more complex computational tasks, accelerating the development of AI technologies. The introduction of recurrent neural networks (RNNs) notably enhanced natural language processing capabilities, allowing machines to understand and express textual content with greater precision.
This technological progress led to the birth of modern intelligent assistants, which not only comprehend human language but also recognize user needs and perform corresponding actions to fulfill those needs. These intelligent assistants can respond to inquiries based on pre-set scripts or integrate with external services to provide more personalized assistance.
A New Era for Generative AI
Today, we find ourselves in a new era of generative AI technology. Generative AI is a branch of deep learning that relies on an advanced model architecture known as the Transformer. This architecture surpasses traditional RNNs, allowing for the processing of longer text sequences and more effectively capturing relationships within the text.
The Transformer model employs a technique called attention mechanisms, enabling AI to accurately identify crucial information within the input data and generate responses based on that information. This means that generative AI can not only understand your input but also creatively generate text, images, or other content.
How do large language models work?
In the next section, we will explore the various types of generative AI models. However, let us first examine how large language models, specifically the OpenAI GPT (Generative Pre-trained Transformer) model, operate.
- Tokenization: From Text to Numbers
Large language models accept text as input and generate text as output. However, as statistical models, they are far more efficient at processing numbers than text. This is why the input must first be processed by a tokenizer before being used by the model. A token is a segment of text composed of a variable number of characters, and the primary task of the tokenizer is to split the input into an array of tokens. Each token is then mapped to a token index, which serves as the integer encoding of the original text segment.
-
Predicting Output Tokens
When the model receives n tokens as input (where n is limited by the model’s maximum capacity), it predicts a single token as output. This token is then added to the input for the next iteration in an expanding window fashion, facilitating a more coherent user experience, such as generating one or more complete sentences. This is why you may notice that ChatGPT sometimes pauses in the middle of a sentence. -
Selection Process and Probability Distribution
The model selects the output token based on the probabilities of subsequent tokens in the current text sequence. It predicts a probability distribution for all potential “next tokens.” However, the model does not always choose the token with the highest probability. To introduce a degree of randomness and enhance the creativity of the model’s output, the selection process is not entirely deterministic. As a result, the same input does not always yield identical outputs. This randomness is regulated by a parameter known as “temperature”; a higher temperature value results in more random outputs.
How Can We Utilize Large Language Models?
Now that we have a better understanding of the inner workings of large language models, let us explore some of their practical applications in common tasks.
The primary capability of large language models lies in their ability to generate text from natural language input. So, what do the inputs and outputs look like? The input for large language models is referred to as a prompt, while the output is known as a completion, which refers to the model’s mechanism for generating the next token to complete the current input. We will delve deeper into what constitutes a prompt and how to design prompts effectively to maximize the model’s potential. For now, we can simply state that a prompt may include:
- Instructions that specify the type of output we expect the model to generate. These instructions may sometimes include examples or additional data.
-
- Summaries of articles, books, product reviews, and other content, extracting insights from unstructured data.
-
- Creative conception and design of articles, papers, assignments, etc.
- Questions in the form of a conversation with an agent.
- A paragraph that needs to be completed, with an implicit request for writing help.
- Ask to explain and annotate a piece of code, or ask to generate a code snippet that can perform a specific task.
The examples provided above are rather simple and only showcase some of the advantages of using generative artificial intelligence, without thoroughly demonstrating the full capabilities of large language models.
Moreover, the outputs of generative AI models are not flawless; at times, the model’s creativity may backfire, resulting in a combination of words that human users might interpret as a confusion of reality or, in some cases, potentially offensive content. Generative AI does not possess intelligence—at least not in the broader sense that encompasses critical and creative reasoning or emotional intelligence. It is not deterministic and cannot be entirely trusted, as fabricated content, such as incorrect citations, information, and statements, may be presented alongside accurate information in a persuasive and confident manner. In the upcoming lessons, we will address all these limitations and explore what measures we can take to mitigate these issues.
Conclusion
Through this chapter, we have gained a preliminary understanding of the fundamental concepts of generative AI, its historical development, and its profound impact on our programming world. The rise of generative AI presents us with unprecedented opportunities, allowing us to interact with machines in a more natural and intuitive manner, leveraging their intelligence to realize our creative visions. As we continue our journey through the “Introduction to Generative AI Development” course, you will learn how to apply these technologies to develop intelligent dialogue systems, image generation tools, and content creation applications, transforming theoretical knowledge into practical applications while staying at the forefront of technology. Let us embark on this exciting learning journey together and explore the limitless potential of generative AI!