TensorBoard - Our Batman for DNNs

Updated on •

categories - machine_learning

tags - tensorflow, tensorboard


So, the first tensorflow dev summit was over, and the biggest news was Version 1.0, with all these new features. One of them which caught my eye was TensorBoard. Well the speaker summarized it best as -
If neural networks are Black box, then Tensorboard is Flashlight.
Overview
In this tutorial I will present some things that I can make tensorboard do ;). If you know about tensorboard, and are here to code, skip this section.So, lets talk more about tensorboard. Tensorboard is a tool that can help us visualize summaries of variables, the graph, and embeddings (t-SNE, and PCA). And to make it do that, we have to follow these steps -
  1. Make a FileWriter Object. It is responsible for writing data onto hard disks.
  2. Use tf.summary module for recording different summaries.
  3. Make a tensor of merged summaries, using merge_all_summaries method, and then evaluate using the session object at some intervals in training.
  4. Write that summary to the filewrite object using add_summary method.
  5. Write out the graph to the object using add_graph method.
Lets get started
So with that, lets begin to try our hands on the tool. Since while writing, I am a newbie to tensorflow myself, I would start with simplest tensor computations and then proceed to a proper neural network. Lets start with the following code, which has 2 layers, one of them calculates the mean and also multiplies the input vector by 10, and nested layer calculates the sum of the product.
Code Results
You can see the results using the command tensorboard --logdir=/path in the code Tensorboard is already installed with tensorflow and it generally doesn't produce an error if wrong path is given in the command, so verify accordingly. You can navigate to localhost:6006 to view results yourself.
Generated graph
Scalar summaries
Some more Info
With this I think we get a pretty nice idea of what's happening under the hood. We name the variables, we get the results. Now what are all the type of data that can be displayed or kept track of, should be our concern. I think it would be pretty interesting to see the mean of weights during a training of model, and we can do that just the way we did now. Below is a list of possible summaries that can be written -
  1. Scalars - One valued constants.
  2. Audio
  3. Image
  4. Histogram
One complete Example
This would be the final example, an I would like to cover as many aspects of the module as possible. Also since I am closely following the Stanford cs224n course(2017 version), the coding style and moreover the format is highly related to that. It might seem a bit fuzzy at once, but it is maintable code :). I would like to highlight some practices I think would be good to maintain for summaries and training models, like during epochs, record them, in cycle of 5 and others, still, leave feedbacks for improvements. Check the code in this blog post (here(in draft, not published)). As this code is part of my semester project for Cryptography N/W security me :). It is explained there in different context, but you can also see the summary recording that I did for each variable. Here are some practices that I would generally use -
  1. For most type of summaries, adding this (gist) code would be good. Any type of tensor would have these details in them.
  2. Nest the layers into name scopes as much as you can. This would organize the visualized graph and would also help you combine multiple summaries using regex feature.
  3. For the batch you are writing summary, it would be good to use a validation set as your feed_dict into session object. What I would like to do is switch between them on the ratio of dataset size of both.
  4. What a cool thing that I liked was the dev summit demonstration of Hyperparameter optimization during multiple runs. Check out the video in the links section, it's really awesome
  5. Will add more as I come through..

Links

  1. Youtube Video
  2. TensorFlow Graph Visuals