{"id":525,"date":"2023-06-12T16:38:08","date_gmt":"2023-06-12T07:38:08","guid":{"rendered":"https:\/\/skanto.co.kr\/?p=525"},"modified":"2023-06-12T16:38:08","modified_gmt":"2023-06-12T07:38:08","slug":"deep-learning-neural-network-in-tensorflow","status":"publish","type":"post","link":"https:\/\/skanto.co.kr\/?p=525","title":{"rendered":"Deep Learning: Neural Network in TensorFlow"},"content":{"rendered":"\n<h2 class=\"wp-block-heading\">Model<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Each pair of input data and the desired answer is called an <em>example<\/em>.<\/li>\n\n\n\n<li>With the help of the examples, the training process produces the automatically discovered<br><em>rules<\/em>.<\/li>\n\n\n\n<li>A human engineer provides a blueprint for the rules at the outside of training. The<br>blueprint is encapsulated in a <em>model<\/em> which forms a <em>hypothesis space<\/em> for the rules the<br>machine may possibly learn.<\/li>\n\n\n\n<li>Models vary in terms of how many layers the neural netowrk consists of, what types of<br>layers they are, and how they are wired together.<\/li>\n\n\n\n<li>With the training data and the model architecture, <strong>the training process produces the learned rules, encapsulated in a trained model<\/strong>.<\/li>\n\n\n\n<li>Training Phase -&gt; Inference Phase<\/li>\n<\/ul>\n\n\n\n<h1 class=\"wp-block-heading\">Neural network and Deep learning<\/h1>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Neural networks are a subfield of machine learning, one in which the transformation of<br>the data representation is done by a system with an architecture loosely inspired by<br>how neurons are connected in human and animal brains.<\/li>\n\n\n\n<li>A frequently encountered theme of neuronal connection is <strong>the layer organization<\/strong>.<br>Many parts of the mammalian brain are organized in a layered fashion. Examples include<br>the retina, the cerebral cortex(\ub300\ub1cc\ud53c\uc9c8), and the cerebellar cortex(\uc18c\ub1cc\ud53c\uc9c8).<\/li>\n\n\n\n<li>Neural network layers are different from pure mathematical functions in that they are generally <em>stateful<\/em>.\n<ul class=\"wp-block-list\">\n<li>layer&#8217;s memory is captured in its <em>weights<\/em>.<\/li>\n\n\n\n<li><strong>weight<\/strong>: a set of numerical values that belong to the layer and govern the details of<br>how each input representation is transformed by the layer into an output<br>representation.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>When a neural network is trained through exposure to training data, the weights get<br>altered systematically in a way that minimizes a certain value called the <em>loss function<\/em>.<\/li>\n\n\n\n<li>Generally, <strong>backpropagation<\/strong> in a neural network computes <em>the gradient of the loss function<\/em> with respect to <em>the weights of the network for single input or output<\/em>.<\/li>\n\n\n\n<li>Basically, a <strong>dense layer<\/strong> is used for changing the dimension of the vectors by using<br>every neuron.<\/li>\n\n\n\n<li><strong>Activation<\/strong>: In neural networks, the activation function is a function that is used for<br>the transformation of the input values of neurons. Basically, it introduces the<br>non-linearity into the networks of neural networks so that the networks can learn the<br>relationship between the input and output values.<\/li>\n\n\n\n<li><strong>Deep Learning<\/strong> is the study and application of deep neural networks, which are, quite<br>simply, neural networks with many layers(typically, from a dozon to hundreds of layers)<\/li>\n\n\n\n<li>Deep learning(layered representation learning) vs. Shallow learning<\/li>\n\n\n\n<li><strong>Feature Engineering<\/strong>\n<ul class=\"wp-block-list\">\n<li>Deep learning automates this features engineering<\/li>\n\n\n\n<li>with deep learning, you learn all features in one pass rather than having to engineer them yourself.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Two essential characteristics<\/strong>\n<ul class=\"wp-block-list\">\n<li><strong>the incremental<\/strong>, layer-by-layer way in which increasingly complex representations are developed<\/li>\n\n\n\n<li>the fact that <strong>these intermediate incremental representations are learned jointly<\/strong>, each<br>layer being updated to folow both the representatioinal needs of the layer above and the needs<br>of the layer below.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>CUDA(2007): Computer Unified Device Architecture<\/li>\n\n\n\n<li>If hardware and algorithms are the steam engine of the deep-learning revolution, then<br>data is its coal.<\/li>\n\n\n\n<li><strong>TensorFlow<\/strong> was made open source in <strong>November 2015<\/strong> by a team of engineers working on deep learning at Google.\n<ul class=\"wp-block-list\">\n<li>data representations called <em>tensors<\/em> flow through layers and other data-processing nodes,<br>allowing inference and training to happen on machine-learning models.<\/li>\n\n\n\n<li>tensor: multidimensional array<br>: In neural networks and deep learning, every piece of data and every computation result<br>is represented as a tensor.<\/li>\n\n\n\n<li>Each tensor has two basic properties: the data type(such as float32 or int32) and the shape<\/li>\n\n\n\n<li>The tensor is the lingua franca of deep-learning models.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>Tensorflow and Keras form an ecosystem that leads the field of deep-learning frameworks in<br>terms of industrial and academic adoption.<\/li>\n\n\n\n<li>deeplearn.js: released 2017.09<\/li>\n<\/ul>\n\n\n\n<h1 class=\"wp-block-heading\">Layer: a data processing module<\/h1>\n\n\n\n<ul class=\"wp-block-list\">\n<li>You can think of as a tunable function from tensors to tensors.<\/li>\n<\/ul>\n\n\n\n<h1 class=\"wp-block-heading\">the kernel and bias = weights<\/h1>\n\n\n\n<ul class=\"wp-block-list\">\n<li>To find a good setting for the kernel and bias we need two things\n<ol class=\"wp-block-list\">\n<li>a measure that tells us how well we are doing<\/li>\n\n\n\n<li>a method to update the weights&#8217; values that next time we will do better than we currently are doing<br>according to the measure previously mentioned.<\/li>\n<\/ol>\n<\/li>\n<\/ul>\n\n\n\n<h1 class=\"wp-block-heading\">model compilation<\/h1>\n\n\n\n<ul class=\"wp-block-list\">\n<li>a loss function: an error measurement<\/li>\n\n\n\n<li>an optimizer: the algorithm by which the network will update its weights (kernel and bias) based on<br>the data and the loss function<\/li>\n<\/ul>\n\n\n\n<h1 class=\"wp-block-heading\">epoch<\/h1>\n\n\n\n<ul class=\"wp-block-list\">\n<li>each iteration through the complete training set is called an <strong><em>epoch<\/em><\/strong><\/li>\n<\/ul>\n\n\n\n<h1 class=\"wp-block-heading\">model&#8217;s evaluate method<\/h1>\n\n\n\n<ul class=\"wp-block-list\">\n<li>it is similar to the fit() method in that it calculates the same loss, but evaluate() does not update<br>the model&#8217;s weights.<\/li>\n<\/ul>\n\n\n\n<h1 class=\"wp-block-heading\">backpropagation<\/h1>\n\n\n\n<ul class=\"wp-block-list\">\n<li>The directions are critical to the neural network&#8217;s learning process. They are determined<br>by the gradients with respect to the weights and the algorithm for computing the gradients<br>is called <em>backpropagation<\/em><\/li>\n\n\n\n<li>Invented in the 1960s<\/li>\n\n\n\n<li>is one of the foundations of neural networks and deep learning.<\/li>\n<\/ul>\n\n\n\n<h1 class=\"wp-block-heading\">gradient of loss<\/h1>\n\n\n\n<ul class=\"wp-block-list\">\n<li>y&#8217; = v * x<\/li>\n\n\n\n<li>loss = square(y&#8217; = y) = square(x * x &#8211; y)<\/li>\n\n\n\n<li>how much change in the loss will we get if v is increased by a unit amount<\/li>\n<\/ul>\n\n\n\n<h1 class=\"wp-block-heading\">Why do we need gradient?<\/h1>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Because once we have it, we can alter v in the direction opposite to it, so we can get a decrease<br>in the loss value.<\/li>\n<\/ul>\n\n\n\n<h1 class=\"wp-block-heading\">MSE (Mean Squared Error)<\/h1>\n\n\n\n<ul class=\"wp-block-list\">\n<li>If your application might be sensitive to very incorrect outliers, MSE could be better choice than MAE.<\/li>\n\n\n\n<li>Standard transformation or z-score normalization<br>we will scale our features so that they have zero mean and unit standard deviation.<\/li>\n\n\n\n<li>Refer to this site for more information on zero mean and unit standard deviation.<br>https:\/\/stats.stackexchange.com\/questions\/305672\/what-is-unit-standard-deviation<\/li>\n<\/ul>\n\n\n\n<h1 class=\"wp-block-heading\">Adding nonlinearity: Beyond weighted sums<\/h1>\n\n\n\n<ul class=\"wp-block-list\">\n<li>The primary enhancement we will introduce is nonlinearity &#8211; a mapping between input and output that<br>isn&#8217;t a simple weighted sum of hte input&#8217;s elements.<\/li>\n\n\n\n<li><strong>MLP: Multilayer Perceptron<\/strong>\n<ul class=\"wp-block-list\">\n<li>an oft-used term that describes neural network that 1) have a simple topology without loops(what<br>is referred to as feedforward neural networks) and 2) have a least one hidden layer.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>The number of weight parameters for each layer\n<ul class=\"wp-block-list\">\n<li>This is a count of all the individual numbers that make up the layer&#8217;s weights.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>Activation Function\n<ul class=\"wp-block-list\">\n<li>is an element-by-element transform.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>Sigmoid function\n<ul class=\"wp-block-list\">\n<li>is a &#8220;squashing&#8221; nonlinearity, in the sense that it &#8220;squashes&#8221; all real values from -infinity to +infinity<br>into a much smaller range(0 to +1).<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>Model Neural network and Deep learning Layer: a data processing module the kernel and bias = weights model compilation epoch model&#8217;s evaluate method backpropagation gradient of loss Why do we need gradient? MSE (Mean Squared Error) Adding nonlinearity: Beyond weighted sums<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_import_markdown_pro_load_document_selector":0,"_import_markdown_pro_submit_text_textarea":"","footnotes":""},"categories":[14,7],"tags":[48,59],"class_list":["post-525","post","type-post","status-publish","format-standard","hentry","category-sw-development","category-7","tag-ai","tag-deeplearning"],"_links":{"self":[{"href":"https:\/\/skanto.co.kr\/index.php?rest_route=\/wp\/v2\/posts\/525","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/skanto.co.kr\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/skanto.co.kr\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/skanto.co.kr\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/skanto.co.kr\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=525"}],"version-history":[{"count":1,"href":"https:\/\/skanto.co.kr\/index.php?rest_route=\/wp\/v2\/posts\/525\/revisions"}],"predecessor-version":[{"id":526,"href":"https:\/\/skanto.co.kr\/index.php?rest_route=\/wp\/v2\/posts\/525\/revisions\/526"}],"wp:attachment":[{"href":"https:\/\/skanto.co.kr\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=525"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/skanto.co.kr\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=525"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/skanto.co.kr\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=525"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}