As I mentioned a few days ago, I’m using Gulp on a new WordPress project. I like to back up my work every night, and since a lot of WordPress config and customization happens in the WordPress editor and widgets, that means backing up the mysql database as well as the code.
Why not use this newfound tool? Let’s do it.
I did some searching and found Gulp WordPress Backup, but it was overkill for what I wanted. But I saw that it used an npm package named mysqldump, for the export, so I grabbed that and started setting up a new task in gulpfile.js:
Next step: Defining the filename. I just wanted to use today’s date because I intend on running this at the end of each work day. Since gulp is all javascript, this is easy:
Add this to the gulp task and you are good to go!
Make sure you add mysqldump to your project’s package.json, or at least run npm install mysqldump before using!
I’ll admit, unlike the other tutorials where I was able to figure out most of this on my own, I had to mine other examples to figure this out. I primarily followed Mike Bostock’s Pie Chart Update, II, but his commented Arc Tween example was extremely helpful in understanding what d3.interpolate is doing.
First we need to store the beginning angle for each arc. You’ll recall that I’m using two separate arcs, one for the main chart and one for the labels. For about 10 minutes I was trying to figure out why the first update was jumpy but all subsequent ones were smooth. It turns out that we need to store the initial angles for each set of arcs:
Next we need to write two arcTween functions to transition between the two. I followed Mike Bostock’s example for the first one, then adapted it to the label arc, too:
Last we need to include these arcTween() functions into the change() function we wrote before. I commented out the previous updates so you can compare them. The duration is 1/2 a second:
Here is is in action. As always, you can view source to see the fully integrated example:
Last week we made a pie chart with D3.js. Today we are going to update it in realtime when buttons are clicked.
Here is the basic pie chart again, slightly modified from the original (I only changed the letters):
The key to making this work is D3’s object constancy. We already baked that into the original design by specifying a key function for the underlying presses count data.
Setting Up
First, we need to copy the pie chart we made last week. Make a few updates: change all the presses in data to 1 and change the letters to A, B, and C.
Next, we need something to click so we can increment the count and update the chart. Three buttons will do nicely:
Recomputing the angles
We ultimately want to call a specific update function when we click each button, so let’s write one. We need to consider what the most critical parts of making the original pie chart were so that we can recreate only the necessary steps:
Defining the value function for d3.pie
Computing the angles based on that data
Putting that information in a path so it can be displayed
Remember that we have two arcs: The main pie and the one holding the labels. We need to update both!
Updating the data and chart on click
Now we need to update the underlying data and the whole chart on click:
Here is the result. Click the buttons and watch the chart change!
This is part of my ongoing effort to relearn D3.js. This short tutorial applies what I’ve learned about data joins, arcs, and labels. It varies slightly from other examples like Mike Bostock’s Pie Chart block because I’m using D3.js version 4 and the API for arcs and pies is different from version 3, which most tutorials are based on. I figured this out through some trial and error. Always read the documentation!
I’m eventually going to use this pie chart as a base to learn how to update charts in real time based on interactions like button pushes or clicks.
Getting Started
Always include the library:
Make a div to hold the chart:
D3 stands for Data-Driven Documents. So what do we always start with when creating something with D3? Data!
Since I’m eventually going to figure out how to update this chart in real time based on button presses, I’m going to start with some dummy data for three easy-to-press buttons: q, w, and e. As always, I log it to the console for quick debugging:
We’ll also need some basics like width, height, and radius since we are dealing with a circle here. Make them variables so your code is reusable later:
Next we need a color scheme. Be referring to the API, we learn that we should use .scaleOrdinal() for this:
Setting up the pie and arcs
We need to set up the pie based on our data. According to the documentation, d3.pie() computes the necessary angles based on data to generate a pie or doughnut chart, but does not make shapes directly. We need to use an arc generator for that.
Before we create the SVG and join data with shapes, let’s define some arguments for the two arcs we want: The main arc (for the chart) and the arc to hold the labels. We need an inner and outer radius for each. If you change the inner radius to any number greater than 0 on the main arc you’ll get a doughnut.
Making the shapes
We always start with an SVG. Select the div we created to hold the chart, append an SVG, give the SVG the attributes defined above, and create a group to hold the arcs. Don’t forget to move the center points, or else the chart will be centered in the upper right corner:
Now let’s join the data generated by .pie() with the arcs to generate the necessary groups to hold the upcoming paths. Give them the class “arc”.
Now we can append the paths created by the .arc() functions with the variables we defined above. We’re using the color variable we defined above to get the colors we want for the various arcs:
Once you save, you should see a chart. Now we’re cooking with data!
Labels
Let’s put some labels on it now. We’ll need to append some text tags in each arc, set the position with a transform defined by the labelArc variable we defined earlier, then access the correct letter to add to the label. Then we’ll make it white so it shows up a little better:
There you have it! A basic pie chart. Play around with the variables so you understand better what is going on.
I’ve been on a mission to relearn the fundamentals of D3.js from the ground up. This tutorial is a way to apply what I learned about data joins, click events, and selections. Along the way I learned about building arrays.
I also wrote this up as a block for those interested.
Basics
We want to make a 10×10 grid using D3.js. D3’s strength is transforming DOM elements using data. This means we’ll need some data and we’ll want to use SVG and rect elements.
Data
We could write an array of data for the grid by hand, but we wouldn’t learn anything then, would we? Let’s generate one with Javascript.
Picture a grid in your head. It is made up of rows and columns of squares. Since this is ultimately going to be represented by an SVG, let’s think about how an SVG is structured:
What you see here is a basic structure of rows and columns. That means that when we make our data array, we want to make a nested array of rows and cells/columns inside those rows. We’ll need to use iteration to do this. Easy-peasy.
The other question we’ll have when making these arrays is, “What attributes will this grid need?”. Think about how you’d draw a grid: You start in the upper right corner of a piece of paper, draw a 1×1 square, move over the width of 1 square and draw another, and repeat until you get to the end of the row. Then you’d go back to the first square, draw one underneath it, and repeat the process. Here we’ve described positions, widths, and heights. In SVG world these are x, y, width, and height.
Here is the function I’m using to create the underlying data for the upcoming grid. It makes an array that holds 10 other arrays, which each hold 10 values:
Making a Grid with D3 Data Joins
We made a cool array above and now we’ll make the data correspond to svg:rect objects to make our grid. First we’ll need to make a div to append everything to (and, of course, don’t forget to include the latest version of D3 in your header):
Now we need to assign our data to a variable so we can access it:
Next, let’s append an SVG to the div we made and set its width and height attributes:
Next, we can apply what we learned in Mike Bostock’s Thinking With Joins to make our rows:
And finally we make the individual cells/columns. Translating the data is a bit trickier, but the key is understanding that we are doing a selectAll on the rows, which means that any reference to data is to the contents of the single array that is bound to that row. We’ll then use a key function to access the attributes we defined (x, y, width, height):
You’ll note that I added style fill and stroke attributes to make the grid visible.
When we put it all together, here is what we get:
Note: If you are viewing this on your phone, you might want to switch over to a tablet or desktop. I haven’t optimized this example for mobile because that will needlessly complicate it.
Cool, huh? Go ahead and inspect the element and marvel at your find handiwork. Then change the fill, stroke, width, and height attributes and see how it changes.
Adding Click Functions
Let’s have some fun and add click events to the individual cells. I want to have cells turn blue on the first click, orange on the second, grey on the third, and white again on the fourth. Since D3 is data-driven, we’ll need to add some click data to the arrays and then add functions to change it and set colors based on the number of clicks.
Let’s break down that on('click') function:
When you click on a cell, it increases the click variable (originally set at 0) by 1.
The if statements set the color based on how many times it has been clicked mod 4. This satisfies the UI of only having four states: white, blue, orange, and grey. If you go to your console and call up the data for a certain cell, you’ll see that the full number of clicks is available.
Here it is. Click away!
Randomized click counts
What happens when we randomize click counts when we create the data array?
Note that Math.random() returns a number between 0 and 1, inclusive. Multiple that by 100 if you want a number between 1 and 100.
It changes when you refresh!
Mouseovers: Even more fun with a bigger grid
What happens when we change the click event to a mouseover event and make a bigger grid? It becomes a lot more fun. I’ll leave the implementation as an exercise to the reader. If you’ve been following along and writing this yourself instead of copying and pasting, you probably already know which variables and events to change:
These are my outputs from Mike Bostock’s Three Little Circles Tutorial, slightly modified so I could understand how each of these items work. This is a tutorial about selections and basic data joins.
We start with three basic SVG circles:
Now let’s select them and make them a little bigger and orange!
Now how about making them light blue and having a randomly-generated height that resets every second?
That’s cool!
Now let’s do some data joins. How about making the radius of the circle a function of a data set?
Looks like the radius is a square of the data. Go ahead and inspect the element to confirm!
You’ll notice that the function uses the name d to refer to bound data.
Let’t make these purple for variety and write a linear function to space them out horizontally.
This uses a new function: The index of the element within its selection. The index is often useful for positioning elements sequentially. We call it i by convention.
D3.js is a javascript library for manipulating documents based on data. It uses HTML, SVG, and CSS to do so. It emphasizes web standards so that you can use modern browsers without proprietary frameworks.
You can bind data to a DOM (Document Object Model) and apply data-driven transformations to the document.
Example: Generate an HTML table from an array of numbers, then use the same data to make an SVG bar chart.
Selections
D3 makes it a lot easier to modify documents than the W3C DOM API. The W3C method relies on verbose names and manual iteration. D3 employs a declarative approach that operates on arbitrary nodes called selections.
Here is Mike Bostock’s examples for how to change the text color of paragraph elements with the W3C method vs D3 method:
W3C:
D3:
Elements may be selected using a variety of predicates, including containment, attribute values, class and ID.
D3 provides multiple ways to change nodes: Setting attributes and styles, registering event listeners, adding/removing/sorting nodes, and changing HTML or text content.
Direct access to the DOM is possible because each D3 selection is a simple array of nodes.
Dynamic Properties
D3 has similar syntax to jQuery, but styles, attributes, and other properties can be specified as functions of data in D3, not just constants.
To alternate shades of gray for even and odd nodes:
Computed properties often refer to bound data. Data is specified as an array of values, and each value is passed as the first argument (d) to selection functions. With the default join-by-index, the first element in the data array is passed to the first node in the selection, the second element to the second node, and so on. For example, if you bind an array of numbers to paragraph elements, you can use these numbers to compute dynamic font sizes:
Once the data has been bound to the document, you can omit the data operator. D3 will retrieve the previously-bound data. This allows you to recompute properties without rebinding.
Enter and Exit
With D3’s enter and exit selections you can create new nodes for incoming data and remove outgoing nodes that are no longer needed.
When data is bound to a selection, each element in the data array is paired with the corresponding node in the selection. If there are fewer nodes than data, the extra data elements form the enter selection, which you can bring to life by using the .enter() selection. Example:
Updating nodes are the result of the data operator. If you forget the enter and exit selections, you will automatically select only the elements for which there exists corresponding data.
A common pattern is to break the initial selection into three parts: Updating nodes to modify, entering the nodes to add, and exiting the nodes to remove. Example:
By handling the three cases (update, enter, exit) separately, you control precisely which operations run on which nodes.
D3 allows you to transform documents based on data. This includes creating and destroying elements.
D3 allows you to change an existing document in response to user interaction, animation over time, or even asynchronous inputs from a third-party. A hybrid approach is also possible, where the document is initially rendered on the server and updated on the client via D3.
Transformation, not Representation
D3 does not introduce a new visual representation. Instead, its graphical marks come from web standards: HTML, SVG, and CSS. If browsers introduce new features tomorrow, you can use them immediately with D3, no update required.
D3 is easy to debug with the browser’s built-in element inspector. The nodes D2 manipulates are the same ones the browser uses natively.
Transitions
D3’s transformation focus extends to animated transitions, too. Transitions interpolate styles and attributes over time. The time between can be controlled with easing functions.
D3’s interpolators support primitives (numbers and numbers within strings like font size, etc) and compound values. They are also extendable.
Examples:
To fade the background of the page to black:
Or, to resize circles in a symbol map with a staggered delay:
D3 allows you to modify only the elements that change, which reduces overhead and allows more complexity at high frame rates.
D3 allows sequencing of complex transitions via events
D3 does not replace the browser’s toolbox, but instead exposes it and makes it easier to use. You can still use CSS transitions.
D3 has no primitive for creating multiple DOM elements. The .append() method can create single elements, but if you want multiple, you need to think in a different way.
Instead of telling D3 how to do something, tell D3 what you want. For example, on the Thinking with Joins page, Mike uses this snippet to tell D3 that the selection “circle” should correspond to data points. This concept is called a data join:
Data points joined to the existing elements produce the update (inner) selection. Leftover unbound data produce the enter (left) selection, which represents missing elements. Any remaining unbound elements produce the exit (right) selection, which represents elements to be removed.
Now let’s explain the svg.selectAll("circle"):
First it returns an empty selection since the SVG container was empty. The parent node of the selection was the SVG container.
The selection is joined to a data array, resulting in three new selections that represent three possible states: enter, update, or exit. Since the selection was empty, the update and exit selections are empty and the enter selection contains a placeholder for each new data point.
The update selection is returned by selection.data and the enter and exit selections hang off the update selection. selection.enter returns the enter selection.
The missing elements are added to the SVG container by calling selection.append on the enter selection. This appends a new circle for each data point in the SVG container.
Thinking with joins means declaring a relationship between a selection (such as a “circle”) and data, then implementing this relationship through the three enter, update, and exit states.
For static visualizations, the enter selection is sufficient. But you can support dynamic visualizations with minor modifications to update and exit.
If a given enter, update, or exit selection happens to be empty, the corresponding code does not operate.
Joins let you target operations to specific states. For example, you can set constant attributes (such as the circle’s radius, defined by the “r” attribute) on enter rather than update. By reselecting elements and minimizing DOM changes, you vastly improve rendering performance!
Similarly, it allows you to target animations to specific states like expanding circles as the come in and contract circles as they go out.
Here is a list of resources I plan to go through. I should be able to relearn the basics fairly quickly so that I can spend time making charts of my own. I’ll update completed items with check marks as I go through them.
Letâs say you have a bunch of elements that under certain situations you want to replace with another element. jQueryâs replaceWith() function is perfect for that.
Here is an example of replacing all links with the class replace with a different link. The use case was finding certain buttons that say âApply Nowâ and replacing them with buttons that say âFinish your applicationâ. Since searching for âApply Nowâ wasnât perfect (the text and other links said it, too) I manually put in the replace class so I knew exactly which elements Iâd be selecting.
Want to perform this action if a cookie exists? Try using js-cookie:
The basic way searching works is that you create an index of all content on a site you want to search, then the search query is checked against that index. On something like a WordPress site, this query is checked against the database, which is a defacto index. Google does its own indexing of your site, so the Google queries are checked against Googleâs index (likely also stored in a database).
Since Jekyll sites are static (meaning that they are only files, no databases included) there is no database to search. That leaves us with two options:
Create an index on your own and make sure your search queries check against that index.
I love Google CSE, but I decided to go with the second route this time. I used Mat Haywardâs Jekyll Search scripts to do the heavy lifting, but I edited them to suit my needs. It is lightning fast!
These scripts:
Generate a JSON file of my site contents when the site is built
Compare the search query to the contents of the JSON file
Return the results based on a template
I modified the way the dates are formatted, how the excerpt gets put into the JSON file, the output styles, and put in a conditional for external link posts.
Let’s say you have two forms: One is a basic contact info form and the other is more in-depth. You have them separated because you want to capture contact information as soon as possible to follow up on leads, but you want some information to be carried over to the second form. You also want the user to be greeted with a message to finish the form when they come back to the site (if they leave).
Here is a sketch of what you’d need to do:
When the first form is filled out and submitted, set a cookie. You could either use vanilla Javascript or a plugin like js-cookie.
The cookies should contain the value of the form fields that were entered. Use something like form.fieldname.value in JavaScript or $("input").val() in jQuery to retrieve them.
Set the cookies for a reasonable amount of time. 2 weeks?
On the landing page, check for cookies. If they are there, change the call to action from “Apply Now” to “Finish your application!”
On the second form page, write a function to check to see if there is are cookies set. If there is, read the cookies and insert their values into the form.
When the form on the second page is submitted, use a function to remove the cookies. They are no longer necessary.
The implementation is left as an exercise for the reader. This outline should get you well on your way!
Today I learned about jQuery’s greater than selector, :gt().
The :gt() selector selects elements with an index number higher than a specified number. The index numbers start at 0.
I’m using this selector to hide more than 10 recent posts on my TIL page. Each of these posts is inside a list item (li) and a child of the unordered list (ul) with a class called “recent”. I then added a toggle function on a button to show them again by using the same selector.
Example:
There is also a less than selector in jQuery. It works similarly, except that it selects elements with an index number lower than the specified number. Syntax: :lt()
Note: Since I’m using a Jekyll site and generating that TIL list via Liquid, another solution is to set a counter after which it automatically adds a class directly to the HTML to hide those items that we can then toggle later. I think the jQuery solution is cleaner, though.
Trifacta Wrangler is a free program (currently in beta) that helps you clean up data sets and gives you a first cut at basic analysis. It is great for quickly turning messy data into structured, manageable formats.
In the past few days I’ve used it to analyze huge log files and turn messy JSON in structured CSVs that I could import into SQL.
Quick tips:
splitrows always has to come first. The program usually tries to split by \n (new line) first, but that doesn’t always work for JSON. Try splitting by something like },{, or do a quick find and replace ( },{ for }|||{ ) and do the split by ||| if you want to keep the curly brackets for an unnest.
unnest is very powerful for splitting out JSON values out into separate columns titled by their keys.
flatten works better than unnest in cases where the JSON does not have keys. It creates new rows and repeats other values in adjacent columns to keep the relation. This works well if you have an ID column and are going to eventually stuff things into a relational database.
Data scientists notoriously hate word clouds. Besides for figuring out what the top 2-3 words are (because they are the biggest), it is difficult to see how much one word is used relative to another. Unfortunately, clients and non-data people love word clouds and sometimes insist on them. What is a self-respecting data nerd to do?
Pair it a word frequency chart!
The easiest way to do this is by using Python’s counter:
Then you can use your favorite charting tool to make a bar chart of the results. I prefer D3.js.
Results
If you see both together, you get a better understanding of the words being used. Of course, a single word doesn’t always capture sentiment. They can be helpful in smaller data sets, but sometimes common phrases are more helpful in larger data sets. For common phrases, use n-gram analysis.
Ember.js is an open-source Javascript framework with the Model–view–viewmodel pattern.
Ember is an opinionated framework. This means that most architectural design decisions have been made for you by the developers of the framework. The advantage of this is that anyone who knows Ember can load your code and understand within a few minutes what is going on.
Core Concepts
Ember router maps a URL to a route handler
Route handler renders a template, then a model that is available to the template. Templates use Handlebars syntax.
Models save data in a “persistent state”, which is fancy language for putting it in a database or data store of some kind.
Components control how the UI behaves. They have two parts: A handlebars template and a javascript source file that defines the behavior.
Installing and running a project
Install:
Creating a new app:
Starting the development server. You must be in the project folder (cd to it). It will serve on http://localhost:4200/
Jekyll has a built-in filter called jsonify to turn data into JSON and Liquid has a filter called json. These is pretty useful for turning standard Jekyll output arrays into JSON, but if you need something more customized, you’ll need to build it yourself because Liquid’s ability to create arrays is limited. (Not a criticism, just an observation. It is primarily a templating language.)
For example, I wanted to store the dates I have blog posts for and the number of posts on each of those dates in order to pass it to a heatmap calendar. I couldn’t figure out a way to build that into a single array with Liquid (and I didn’t want to drop down into Ruby), so I figured out a way build JSON manually.
This outputs my blog’s post dates in the Unix timestamp format and displays the number of posts on a given day and leaves the trailing comma off to comply with the JSON standard:
The key to getting the counts right was advancing the counter by 1, comparing the date of the current post to the previous post, outputting the date and the count if they did not match, and if they did match advancing the counter by 1 and comparing again. After the date and count are output, the counter is reset to 0.
Thanks to Eric Davis for advice on using less code and pointing out how I can make the trailing comma if statement more efficient: %if forloop.last == false% , %endif%
He said:
If there’s one bit of “programming advice” I would give to anybody mucking around with this stuff, it’s this: Write as little code as humanly possible. The most bug-free line of code is the line not written.
What I need to ask myself: If something is being repeated, why?
Here is a minimalist responsive bar chart with quantity labels at the top of each bar and text wrapping of the food labels. It is actually responsive, it doesn’t merely scale the SVG proportionally, it keeps a fixed height and dynamically changes the width.
For simplicity I took the left scale off. All bars are proportional and are labeled anyway.
Go ahead and resize your window! This has a minimum width of about 530px because of the text labels. Any smaller than that and they are very difficult to read.
The basic HTML
The Styles
You’ll see that the axis is actually there but it is white. I found it useful to learn to draw it, but I didn’t want it so I am keeping it hidden.
The Data
The Javascript Heavy Lifting
This is where D3 really comes in.
Setting the margins, sizes, and figuring out the basic scale.
Setting the axes
Drawing the basic SVG container with the proper size and margins
Scaling the axes
Drawing the bars themselves
Adding the quantity labels to the top of each bar
This took me a while to figure out because I was originally appending to the rect element. According to the SVG specs this is illegal, so I moved on to appending them after everything else to they’d show on top. The positioning is tricky, too. I eventually found the correct variables to position it close to center. Then text-anchor: middle; sealed the deal.
Responsiveness
The general method for making D3 charts responsive is to scale the SVG down proportionally as the window gets smaller by manipulating the viewBox and preserveAspectRatio attributes. But after digging around on Github for a while, I found a fancier solution that preserves the height and redraws the SVG as the width shrinks.
Wrapping text labels
Wrapping text labels is tricky. The best solution I found is the one Mike Bostock (D3’s creator) describes. I modified it slightly to work with my chart, but the overall solution is the same.
In the days.animate section, I make sure to prevent the circle from ever reaching 100%. It will get extremely close as the years pass and the numerator comes closer and closer to the denominator, but the denominator will always be slightly larger. Thanks to Eric Davis for helping me come up with the solution.
Javascript Counter with circular progress
The results:
The javascript, which counts the time since a certain date:
The elements it targets:
And the styles to display it:
Next steps: Registering this with a WordPress shortcode.
This isn’t new, but I use it enough that I like to keep a basic snippet handy.
This does the following:
Hides div-1 and div-2 on page load.
Opens div-1 and closes div-2 when button-1 is clicked
Opens div-2 and closes div-1 when button-2 is clicked
See it in action below. Styling the buttons will be left as an exercise to the reader:
This is Div-1!!!
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Phasellus tempus, mauris in sollicitudin commodo, eros mauris euismod arcu, eget faucibus enim tellus a turpis. Suspendisse malesuada interdum elit sit amet rutrum. Nam ut elit lobortis, lobortis nisi sed, tincidunt nibh. In consequat eleifend quam, vestibulum efficitur velit scelerisque nec. Quisque quis rutrum mauris, ut ultricies ipsum. Cras interdum ipsum in diam venenatis fermentum. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia Curae; Donec mollis tincidunt est vitae dapibus. Aenean cursus, ipsum ac pellentesque malesuada, mauris magna pretium nunc, eu mollis dolor ipsum sit amet felis. Vestibulum sed erat eu dolor condimentum viverra blandit sit amet mauris. In arcu nisi, sollicitudin a dui a, laoreet pellentesque libero.
Woah, this is Div-2!~!~!
Suspendisse bibendum sem vitae tellus maximus dictum. Nunc vestibulum tellus sem, sit amet efficitur ligula porta sed. Aliquam erat volutpat. Suspendisse in enim elementum, iaculis magna eu, placerat enim. Aliquam auctor ultricies ligula eleifend sodales. Vestibulum at leo leo. Praesent sollicitudin ipsum ut maximus malesuada.
If you copied this example and it isn’t working, make sure you gave the divs and buttons the correct classes and you included jQuery in your header: