{"id":605,"date":"2023-08-22T22:57:25","date_gmt":"2023-08-22T13:57:25","guid":{"rendered":"https:\/\/skanto.co.kr\/?p=605"},"modified":"2023-08-29T13:52:26","modified_gmt":"2023-08-29T04:52:26","slug":"generative-deep-learning","status":"publish","type":"post","link":"https:\/\/skanto.co.kr\/?p=605","title":{"rendered":"Generative Deep Learning"},"content":{"rendered":"\n

What is Generative Modeling?<\/h2>\n\n\n\n
\n

Generative modeling is a branch of machine learning that involves training a model to produce new data that is similar to a given dataset<\/p>\n<\/blockquote>\n\n\n\n

\"\"
figure 1-1<\/figcaption><\/figure>\n\n\n\n

We can sample from this model to create novel, realistic images of horses that did not exist in the original dataset.<\/p>\n\n\n\n

One data point in the training data is called as observation<\/em>. Each observation consists of many features<\/em>.<\/p>\n\n\n\n

A generative model must be probabilistic<\/em> rather than deterministic<\/span>, because we want to be able to sample many different variations of the output, rather than get the same output every time.<\/p>\n\n\n\n

Generative Versus Descriminative Modeling<\/h3>\n\n\n\n
\"\"
figure 1-2<\/figcaption><\/figure>\n\n\n\n

Figure 1-2 shows the discriminative modeling process.<\/p>\n\n\n\n

Generative modeling doesn’t require the dataset to be labeled because it concerns itself with generating entirely new images, rather than trying to predict a label of a given image.<\/p>\n\n\n\n

\n

Conditional Generative Models<\/h4>\n\n\n\n

For example, if out dataset contains different types of fruit, we could tell our generative model to specifically generate an image of an apple.<\/p>\n<\/blockquote>\n\n\n\n

The Rise of Generative Modeling<\/h3>\n\n\n\n

Until recently, discriminative modeling has been the driving force behind most progress in machine learning. This is because the corresponding generative modeling problem is typically much more difficult to tackle<\/em>.<\/p>\n\n\n\n

However, as machine learning technologies have matured, this assumption has gradually weakened.<\/p>\n\n\n\n

In the last 10 years many of the most interesting advancements in the field have come through novel applications of machine learning to generative modeling tasks.<\/span><\/p>\n\n\n\n

Generative services that target specific business problems<\/p>\n\n\n\n

    \n
  • generate original blog posts given a particular subject matter<\/li>\n\n\n\n
  • produce a variety of images of your product in any setting<\/li>\n\n\n\n
  • write social media content and ad copy to match your brand<\/li>\n\n\n\n
  • game design and cinematography to output video and music<\/li>\n<\/ul>\n\n\n\n

    Generative Modeling and AI<\/h3>\n\n\n\n

    There are three deeper reasons<\/span><\/em> why generative modeling can be considered the key to unlocking a far more sophisticated form of artificial intelligence that goes beyond what discriminative modeling alone can achieve.<\/p>\n\n\n\n

    Firstly<\/strong>, we shouldn’t limit our machine training to simply categorizing data and should also be concerned with training models that capture a more complete understanding of the data distribution. Many of the same techniques that have driven development in discriminative modeling, such as deep learning, can be utilized by generative models too.<\/p>\n\n\n\n

    Secondly<\/strong>, generative modeling is now being used to drive progress in other fields of AI, such as reinforcement learning. A traditional approach(to train a robot to walk across a terrain) is fairly inflexible because it is trained to optimize the policy for one particular task. An alternative approach that has recently gained traction is to train the agent to learn a world model<\/span><\/em> (not the real environment)of the environment using a generative model, independent of any particular task.<\/p>\n\n\n\n

    Finally<\/strong>, if we are to say that we have built a machine that has acquired a form of intelligence, generative modeling must surely be part of the solution. Current neuroscientific theory suggests that our perception of reality is not a highly complex discriminative model operating on our sensory input to produce predictions<\/strong> of what we are experiencing<\/span><\/em>, but is instead a generative model that is trained from birth to produce simulations<\/strong> of our surroundings<\/span><\/em> that accurately match the future.<\/p>\n\n\n\n

    The Generative Modeling Framework<\/h3>\n\n\n\n
    \n
      \n
    • We have a dataset of observation X.<\/li>\n\n\n\n
    • We assume that the observations have been generated according to some unknown distribution, Pdata<\/sub><\/em><\/li>\n\n\n\n
    • We want to build generative model Pmodel<\/sub><\/em> that minics Pdata<\/sub><\/em>. If we achieve this goal, we can sample from Pmodel<\/sub><\/em> to generate observations that appear to have been drawn from Pdata<\/sub><\/em>.<\/li>\n\n\n\n
    • Therefore, the desirable properties of Pmodel<\/sub> are: \n
        \n
      • Accuracy<\/em>
        If Pmodel<\/sub><\/em> is high for a generated observation, it should look like it has been drawn from Pdata<\/sub><\/em>. if Pmodel<\/sub><\/em> is low for a generated observation, it should not look like it has been drawn from Pdata<\/sub><\/em>.<\/li>\n\n\n\n
      • Generation<\/em>
        It should be possible to easily sample a new observation from Pmodel<\/sub><\/em>.<\/li>\n\n\n\n
      • Representation<\/em>
        It should be possible to understand how different high-level features in the data are represented by Pmodel<\/sub><\/em>.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/blockquote>\n\n\n\n

        Representation Learning<\/h3>\n\n\n\n

        Instead of trying to model the high-dimensional sample space directly, we describe each observation in the training set using some lower dimensional latent space<\/strong><\/em> and then learn a mapping function that can take a point in the latent space and map it to a point in the original domain. In other words, each point in the latent space is a representation<\/span><\/em> of some high-dimensional observation.<\/p>\n\n\n\n

        One of the benefits of training models that utilize a latent space is that we can perform operations that affect high-level properties of the image by manipulating its representation vector within the more manageable latent space<\/em>.<\/p>\n\n\n\n

        The concept of encoding the training dataset into a latent space so that we can sample from it and decode the point back to the original domain is common to many generative modeling technique.<\/p>\n\n\n\n

        \"\"
        figure 1-9 The dog manifold in high-dimensional pixel space is mapped to a simpler latent space that can be sampled from<\/figcaption><\/figure>\n\n\n\n

        Core Probability Theory<\/h2>\n\n\n\n

        Five key terms<\/p>\n\n\n\n

        Sample space<\/em><\/p>\n\n\n\n

          \n
        • The sample space is the complete set of all values an observation x can take<\/li>\n<\/ul>\n\n\n\n

          Probability density function<\/em> (or simply density function<\/em>)<\/p>\n\n\n\n

            \n
          • a function p(x)<\/em> that maps a point x in the sample space to a number between 0, and 1. The integral of the density function over all points in the sample space must equal 1, so that it is a well-defined probability distribution.<\/li>\n<\/ul>\n\n\n\n

            Parametric modeling<\/em><\/p>\n\n\n\n

              \n
            • a technique that we can use to structure our approach to finding a suitable Pmodel<\/sub>(x)<\/em><\/li>\n<\/ul>\n\n\n\n

              Likelihood<\/em><\/p>\n\n\n\n

                \n
              • The likelihood \u2112(\u03b8|\ud835\udc31)<\/em> of a parameter set \u03b8 is a function that measures the plausibility of \u03b8 , given some observed point \ud835\udc31<\/li>\n<\/ul>\n\n\n\n

                Maximum likelihood estimation<\/em><\/p>\n\n\n\n

                Generative Model Taxonomy<\/h2>\n\n\n\n

                While all types of generative models ultimately aim to solve the same task, they all take slightly different approaches to modeling the density function. There are three possible approaches:<\/p>\n\n\n\n

                  \n
                1. Explicitly model the density function, but constrain the model in some way, so that the density function is tractable(i.e., it can be calculated).<\/li>\n\n\n\n
                2. Explicitly model a tractable approximation of the density function.<\/li>\n\n\n\n
                3. Implicitly model the density function, through a stochastic process that directly generates data.<\/li>\n<\/ol>\n\n\n\n
                  \"\"
                  figure 1-10 A taxonomy of generative modeling approaches<\/figcaption><\/figure>\n\n\n\n

                  The best-known example of an implicit generative model is a generative adversarial network.<\/p>\n\n\n\n

                  Approximate density models include variational autoencoders.<\/p>\n\n\n\n

                  A common thread that runs through all of the generative model family types is deep learning<\/em><\/p>\n\n\n\n

                  Deep Learning<\/h1>\n\n\n\n

                  Deep Neural Networks<\/h2>\n\n\n\n

                  The majority of deep learning systems are artificial neural networks(ANNs, or just neural networks for short) with multiple stacked hidden layers.<\/p>\n\n\n\n

                  What is a Neural Network?<\/h3>\n\n\n\n

                  A neural network consists of a series of stacked layers<\/em>. Each layer contains units<\/em> that are connected to the previous layer’s units through a set of weights<\/em>.<\/p>\n\n\n\n

                  One of the most common layers is the fully connected<\/em> (or dense) layer that connects all units in the layer directly to every unit in the previous layer.<\/p>\n\n\n\n

                  Neural networks where all adjacent layers are fully connected are called multilayer perceptrons<\/strong><\/em>(MLPs).<\/p>\n\n\n\n

                  \"\"
                  figure 2-2 An example of a multilayer perceptron that predicts if a face is smiling<\/figcaption><\/figure>\n\n\n\n

                  Let’s walk through the network shown in figure 2-2<\/p>\n\n\n\n

                    \n
                  1. Unit A receives the value for an individual channel of an input pixel<\/li>\n\n\n\n
                  2. Unit B combines its input values so that it fires strongest when a particular low-level feature such as an edge is present.<\/li>\n\n\n\n
                  3. Unit C combines the low-level features so that it fires strongest when a higher-level feature such as teeth are seen in the image.<\/li>\n\n\n\n
                  4. Unit D combines the high-level features to that it fires strongest when the person in the original image is smiling.<\/li>\n<\/ol>\n\n\n\n

                    The layers between the input and output layers are called hidden layers<\/em>.<\/p>\n\n\n\n

                    Multilayer Perceptron(MLP)<\/h2>\n\n\n\n

                    The MLP is a discriminative (rather than generative) model, but supervised learning will still play a role in many types of generative models.<\/p>\n\n\n\n

                    Preparing the Data -> Building the Model(Layers, Activation functions) -> Compiling the Model(Loss functions, Optimizers) -> Training the Model -> Evaluating the Model<\/strong><\/em><\/p>\n\n\n\n

                    \n