Improving Jekyll Build Time

When I migrated all of my old posts to Jekyll, my build time skyrocketed. The average time was 39 seconds for 853 posts. Abysmal. This is especially annoying when I’m actively working on the site with jekyll serve and waiting 40 seconds for changes to reflect.

The first step to figuring out what takes so long is running the Liquid profiler: $ jekyll build --profile

This is what it output for me:

Filename                                                | Count |    Bytes |  Time --------------------------------------------------------+-------+----------+------ _layouts/default.html                                   |   853 | 9652.42K | 7.462 search/feed.json                                        |     1 | 1394.16K | 4.148 _includes/header.html                                   |   853 | 1230.35K | 3.485 _layouts/post.html                                      |   755 | 5119.00K | 2.301 _includes/opengraph.html                                |   853 |  870.67K | 1.694 _includes/icons.html                                    |   810 | 1752.89K | 1.454 sitemap.xml                                             |     1 |  203.34K | 1.321 _includes/footer.html                                   |   853 |  109.12K | 0.273 til/posts.yml                                           |     1 |   14.27K | 0.147 about.md                                                |     1 |   18.02K | 0.085 _posts/2016-07-04-posts-heatmap-calendar.md             |     1 |   16.78K | 0.080 isotope.html                                            |     1 |  389.52K | 0.048 _layouts/book.html                                      |    34 |  198.47K | 0.039 til.md                                                  |     1 |   58.76K | 0.023 feed.xml                                                |     1 |  139.11K | 0.022 ... with a dozen more tiny blog index pages from paginator  done in 39.359 seconds. 

I tried a bunch of random stuff that made tiny improvements before I started doing research: Removing extraneous files, cleaning up if blocks that I no longer used, hardcoding things in the header that relied on variables from _config.yml, etc. None of this made significant improvements. I had already moved extraneous plugins long ago and I keep Jekyll up to date, so there was nothing more to do there.

I eventually came across this post by Mike Neumegen, which mentioned jekyll-include-cache, which caches the files you tell it to with the first instance and then serves the cached versions up for subsequent requests. For includes that don’t change based on the page content, you can replace {% include example.html %} with {% include_cached example.html %}. When I did this with my sidebar, nav bar, and footer, it cut my average first build time to 28 seconds, with subsequent regeneration times around 19 seconds. Awesome! (Neumegen’s post, which mentioned jekyll-include-cache contains a lot of other helpful advice for Jekyll build optimization. Check it out!)

My next step was to look at other includes to see if I could shave more time off. Some files had a mix of static and dynamic assets, so I split out all of the static assets into more include_cached includes. This cut another 5-6 seconds off of my build time. Boom.

Now that my layouts and my includes were as optimized as I could get them without major rewrites, I turned to the next major time hog: search/feed.json. 4 seconds for a single file! This file loops through everything on the site and dumps it into a single file so that I can crawl it with Javascript to power search on my site.

I used Mat Hayward’s search project with a few tweaks. As I dug deeper on the specific functions in the feed.json generator, I noticed that everything was being converted from Markdown to HTML first, then HTML stripped out, then turned into JSON. I wondered if skipping the markdown conversion step would lead to inaccurate search, so I tried it. Not only did my search function the same, but it dropped feed.json’s build time down to around a sixth of a second.

My build time now looks like this:

Filename                                                | Count |    Bytes |  Time --------------------------------------------------------+-------+----------+------ _layouts/default.html                                   |   853 | 9564.71K | 2.922 _includes/head.html                                     |   853 | 1571.79K | 2.623 _includes/opengraph.html                                |   853 |  870.67K | 2.171 sitemap.xml                                             |     1 |  203.34K | 1.177 _layouts/post.html                                      |   755 | 5195.80K | 0.509 search/feed.json                                        |     1 | 1394.65K | 0.176 _posts/2016-07-04-posts-heatmap-calendar.md             |     1 |   16.78K | 0.091 about.md                                                |     1 |   18.02K | 0.076 isotope.html                                            |     1 |  389.52K | 0.050 _layouts/book.html                                      |    34 |  198.47K | 0.030 _includes/header.html                                   |     1 |    1.43K | 0.021 til.md                                                  |     1 |   58.76K | 0.019 feed.xml                                                |     1 |  139.24K | 0.018 ... with a dozen more tiny blog index pages from paginator  done in 19.594 seconds. 

Regeneration times when running jekyll serve are hovering around 13 seconds now, which is a significant quality of life improvement for me when managing this blog.

There is probably more work I can do with moving the highlighter to the front end with JS and simplifying the Liquid in head.html and opengraph.html, but I’m stopping here for now.

Troubleshooting Problems Installing Craft CMS on Laravel Homestead

Today I installed Craft CMS in for a new Praxis project. I use Laravel Homestead for my local development environment. It installs like pretty much every other PHP app on Homestead (use this guide by Matt Collins if you aren’t very familiar with Homestead), but I ran into a few annoying errors along the way:

Mcrypt is required

If Mcrypt is required shows when you first go to http://yoursite.local/admin to finish the install, it is probably because you are a good citizen and your Homestead box is actually up to date, running PHP 7.2 as the default. Here’s the issue: mcrypt was removed from PHP 7.2. Craft 2 still needs it. There are three solutions:

  • If you need PHP 7.2, you’ll have to install Craft 3. It is still under development as of this writing, so I didn’t take that path. (Update April 2, 2018 – Apparently Craft 3 is launching on April 4, so you won’t have this issue for long!)
  • You can install the mcrypt extension for PHP 7.2.
  • You can use a different PHP version. I took this route. Homestead makes this super simple by allowing multiple PHP versions. In your Homestead.yaml, set the php variable to php: "7.0" under your site in the Sites block. Here is what mine looks like:
- map: craft-praxis.local      to: /home/vagrant/craft-praxis/public      php: "7.0"

GROUP BY incompatible with sql_mode=only_full_group_by

If you see this message, chances are that you are using MySQL 5.7.5+, which changed the way it deals with GROUP BY. See this StackOverflow post for the two solutions: Removing the ONLY_FULL_GROUP_BY option from sql_mode or adding the initSQLs database config settings in craft/config/db.php.

Building JSON for WPComplete in Excel with Concatenate

I’m using the ImportWP Pro plugin to quickly create pages in the Praxis curriculum portal. One of the custom fields I need to set is for the WPComplete course system we use. It takes JSON for which course the page belongs to and which page to redirect the to once the current page is marked complete. Here is an example:

{"course":"New Module 4 - Thinking","redirect":{"title":"You Have No Idea How Wrong You Are","url":"https://portal.discoverpraxis.com/bootcamp/new-module-4/no-idea-how-wrong"}}

I needed to build this JSON in Excel with the rest of the page info. The pages I’m setting the redirect title and URL for are being created in the spreadsheet, too. So I wanted to build this JSON cell from the contents of other cells. The commas and quotes were a problem for my typical way of building text from other cells: =B5&" "&C5

Here is the new solution I found: Break the JSON down into constituent parts, put each in its own cell, then build the JSON with the concatenate() function.

Here are the 5 cells I needed:

  1. {"course":"New Module 4 - Thinking","redirect":{"title":"
  2. The title from a spreadsheet column
  3. ","url":"https://portal.discoverpraxis.com/bootcamp/new-module-4/
  4. The page slug from a spreadsheet column
  5. "}}

I put 1 in A23, 3 in A24, and 5 in A25. Then pulled 2 and 4 from various cells in columns A and B. Then my JSON cells used this formula: =CONCATENATE(A23,A4,A24,B4,A25)

Concatenated together, it looks like this:

{"course":"New Module 4 - Thinking","redirect":{"title":"PAGETITLE","url":"https://portal.discoverpraxis.com/bootcamp/new-module-4/PAGESLUG"}}

Excel Concatenate

How to Back Up Your Laravel Homestead Databases

Today I upgraded from Homestead 4.0.0 to 5.0.1 so I could test my sites in PHP 7. That was a major upgrade and a number of things changed, so I decided that I needed to back up my databases before I did the upgrade. I’d only ever dumped specific databases before, but TIL how to dump them all at once. Here is how to do that in Laravel Homestead:

  1. $ cd ~/Homestead
  2. $ vagrant ssh
  3. $ mysqldump -u homestead -psecret --all-databases > Code/homestead-YYYY-MM-DD-hh-mm.sql
  4. Test the sql file. I had a syntax error the first time and it only wrote out an error message. You’ve been warned.
  5. Move the SQL file somewhere safe. IE outside of any folder you might delete or change during the upgrade.
  6. After the upgrade, here is how to reload your databases in Homestead: $ mysql -u homestead -psecret < Code/homestead-YYYY-MM-DD-hh-mm.sql

I had some major issues with the upgrade. When I first tried it, vagrant up hung at the SSH auth step and never moved. I went down a deep rabbit hole with my friend Eric Davis trying to debug the SSH issues, to no avail. My last trial was to roll back to Homestead 4.0.0, check my vagrant ssh-config settings, then try the upgrade with those settings. Then, miraculously, when I tried the upgrade again, vagrant up worked with no SSH problems! No difference in settings. I’m baffled, but relieved.

FYI, there are some Homestead.yaml differences between Homestead 4 and 5, so make sure you have some time set aside to read the docs, update your yaml file, and rerun vagrant up --provision a bunch of times to get everything working again.

SVG Viewbox

Today I wrote an article for the Praxis blog outlining four free charting/visualization tools and showing examples of visualizations created with those tools. One outputs SVGs, but they are a fixed size. But since SVGs are just code, I figured I could make it responsive.

First I tried the old responsive image trick: width: 100%; height: auto; – This resulted in the SVG scaling, but only the top 10% was visible.

After some Googling, I came across this article: How to Scale SVG on CSS Tricks. Here I learned about the viewbox attribute. The viewBox does many things:

  • It defines the aspect ratio of the image.
  • It defines how all the lengths and coordinates used inside the SVG should be scaled to fit the total space available.
  • It defines the origin of the SVG coordinate system, the point where x=0 and y=0.

My original SVG was 848px by 500px. So, I set viewbox="0 0 848 500" and width=100% and removed the height attribute.

This led to a nice scalable SVG:

2005200620072008200920102011201220132014201520162017East GateNorth GateNortheast GateSouth GateWest GateHighway 191

CSS Confetti

TIL how to make confetti with CSS. No javascript needed.

I used this codepen as a base and modified it to my needs. The author, @fionnachan, had an interesting method I hadn’t thought of: Make a div for each piece of confetti, transform and skew each one, and then set a random offset so it looks like each one moves independently when you apply keyframes animation. @for in SCSS makes the random offsets and transforms a breeze.

The only downside is that it compiles to almost 4000 lines of CSS. I’m going to try a JS version and see how it performs in comparison.

UPDATE January 7, 2018: I moved to a JS solution based on https://codepen.io/anon/pen/JMOQzE. 50 lines of code + a jQuery dependency that was already getting loaded anyway is much better than 4000 lines of CSS, even if the CSS is novel.

CSS Confetti

Genesis Framework’s Hooks and Filters

TIL about the Genesis framework’s hooks and filters. I’m disappointed in how poorly they are documented. I spent most of my time digging around in the Genesis base theme’s /lib/ folder deciphering their functions and figuring out what powers what via trial and error.

Here are a few resources that helped along the way:

  1. https://genesistutorials.com/visual-hook-guide/
  2. https://my.studiopress.com/documentation/hooks/hooks/hook-reference/
  3. https://my.studiopress.com/documentation/filters/genesis-filters/filter-reference/

Basic Elements of Communication

I listened to most of “A Mind at Play: How Claude Shannon Invented the Information Age” on a road trip today. I learned that all forms of communication share the same basic elements, no matter the sender or medium.

Claude Shannon, the father of information theory, published “A Mathematical Theory of Communication” in 1948, which laid out these basic elements:

  1. A source produces the message.
  2. A transmitter translates the message into a signal that can be sent through a particular medium.
  3. A medium, over which the signal is sent.
  4. Noise interfering with the signal across the medium.
  5. A receiver, which receives the signal and translates it back into a format that the destination can understand.
  6. A destination, for which the message is intended.

Here is a diagram I drew of these elements:

Mathematical Theory of Communication

All forms of communication share these same basic elements, including:

  • Email
  • Tweets
  • Text messages
  • DNA
  • Radio
  • Telephone conversations
  • Traffic signals
  • Television
  • Telegraph
  • Physical letters
  • Carrier Pigeons
  • Smoke signals

Remember to Ask for Help

Yesterday I worked on building the new Praxis website for 12 hours straight, from 12-12, after spending the morning on coaching calls with Praxis participants. I was getting a lot of good work done, then I hit a wall with coding out this timeline:

timeline

I tried a bunch of different solutions for about three hours. Nothing worked. I got frustrated and almost convinced myself that this just couldn’t be done. It was about midnight at this point. I knew how to make the month boxes offset from each other on the left and right and make the whole thing responsive OR make the month boxes overlapping like this and the whole thing static/non-responsive. But not both.

Then, thankfully, I took a step back for a brief moment. The designer we contracted to design this is well-known and a great web developer in his own right. I was pretty sure that he wouldn’t design something that he couldn’t actually build himself. So I emailed him and asked for guidance.

Within a few hours he got back to me with a rough solution. He immediately pointed out two things that alluded me:

  1. Making 3 discrete sizes makes much more sense than making everything completely fluid
  2. The floats work well with content in them, but are completely broken without content in them. I tried to mock this up with very little content in it, which produced strange results. With content in place it functions as expected.

With an hour and a half, I took the rough solution he outlined and turned it into a fully functioning polished version of the design.

This was a great reminder for me:

I don’t know all the answers, even if I think I do.

When I start to get frustrated, I need to take a step back.

Sometimes I need to ask for help.

Responsive Breakpoints in Sass

Up until last week, Sass breakpoints were a bit of a mystery to me. I finally figured out how they work.

First, they are programmatically generated. Define the breakpoints as a map. I use the list here.

$breakpoints: ( 	xs: 512px, 	sm: 768px, 	md: 896px, 	lg: 1152px, 	xl: 1280px );

Next, use a mixin and expressions to build the media queries. I use min-width. I leave it as an exercise for the reader if you wish to reverse this to use max-width instead.

@mixin breakpoint( $breakpoint ) {     @if map-has-key( $breakpoints, $breakpoint ) {         @media ( min-width: #{ map-get( $breakpoints, $breakpoint ) } ) {             @content;         }     } @else if type_of( $breakpoint ) == number and unit( $breakpoint ) == px or unit( $breakpoint ) == em or unit( $breakpoint ) == rem {         @media (min-width: $breakpoint ) {             @content;         }     } @else {         @warn "No value could be retrieved from `#{$breakpoint}`. "                + "It is either not defined in the `$breakpoints` map, or it is an invalid number of px, em, or rem.";     } }

To use, define your breakpoint styles using @import. Here is an example defining H1 font sizes at three different breakpoints:

h1 { 		text-transform: capitalize; 		@include breakpoint(xs) { 				font-size: 2.2em; 		} 		@include breakpoint(sm) { 				font-size: 2.7em; 		} 		@include breakpoint(xl) { 				font-size: 3em; 		} }

Bulk Clearing Expiration Dates in Restrict Content Pro

At Praxis we use Restrict Content Pro as the membership system for our curriculum portal. We decided that all grads get access for life, not just during the program. So, I needed a way to clear over 200 member expiration dates. The only bulk method available through the WordPress interface is to set the expiration dates to another date in the future, which would just kick my problem further down the road. So I needed to dust off my SQL knowledge and directly edit the database.

Step 1: Back up the database.

Don’t be a fool. Back up your database and test the queries on a local development version first. Never run queries for the first time on production. The backup is also a failsafe that you can restore if something goes wrong despite your testing.

Step 2: Find the data.

I saw that all data related to Restrict Content Pro usually had rcp somewhere in the table, column, or key. So I started with the rcp tables. They had nothing to do with expiration dates, so I checked the wp_usermeta table since RCP extends the WordPress users with more functionality. Bingo. There was a column called meta_key with rcp_expiration in it with corresponding date values.

Step 3: Make sure you are editing the correct data by running a SELECT statement first.

Sure, you could run your UPDATE statement first, but I like to make sure I am editing the correct data by running a SELECT statement first and then using the same WHERE clause for my UPDATE statement.

After a few stupid syntax errors, here is the SELECT statement that got exactly what I wanted. This shows the user ID so I can spot check, restricts searching to the rcp_expiration meta key, and looks for values that are not none.

SELECT user_id, meta_key, meta_value FROM wp_usermeta WHERE meta_key = 'rcp_expiration' AND meta_value != 'none';

This returned 176 results. When I changed it to show only meta values that were none, I got 31 values. 31+176=207, which is the total number of users. Looking good.

Step 4: Craft your UPDATE statement.

Now that we know we selected the correct data with our previous statement, it is time to craft our UPDATE statement.

Here I’m updating the wp_usermeta table and setting the meta_value to none where the meta_key is rcp_expiration and the corresponding meta_value is not none.

UPDATE wp_usermeta SET meta_value = 'none' WHERE meta_key = 'rcp_expiration' AND meta_value != 'none';

I tested this on my local machine and it updated 176 rows. Just like we wanted.

Step 5: Run the same query on production.

Now that we’ve tested the query in our development environment and verified that we got the results we wanted, we can run the query on the production database. If you use phpmyadmin and want to triple check that you aren’t messing anything up, you can click the “Simulate Query” button first. (I did.)

Step 6: Verify things worked.

Log in to WordPress and check the RCP membership area. Verify that all expiration dates are now set to none. Also verify that your users can still log in. You should have a few user test logins specifically for this purpose. You can also check your site logs throughout the day to make sure people are still logging in. You can’t count on them always letting you know when something doesn’t work. More often than not they will just stop using it. It is up to you to verify everything works as it should!

Adding Months to a Calculated Date in Salesforce

Adding dates is tricky. Months have different numbers of days, so you can’t rely on just adding 30 days to get an extra month. You also can’t just add a certain number of months because formulas in Salesforce don’t auto increment the year. The solution is modular arithmetic and conditionals.

The goal here was to make a set of fields to send out emails on the first day of each month for 6 months, given a specific month to start with.

What I’m doing here is:

  • Year: Figuring out the month number, adding one less than the number of months over all, dividing it by 12, and rounding down to add either a 0 or 1 to the year. You have to subtract one from the month because 12/12 = 1 and you don’t want December adding an extra year.
  • Month: If the resulting month is December, return 12. Otherwise return the month number modulo 12. (12 mod 12 is 0, hence the conditional).
  • Day: Always returning 01, the first day of the month.
DATE(   YEAR( date ) + FLOOR( ( MONTH ( date ) + number_of_months - 1 ) / 12 ),   IF( MONTH ( date ) + number_of_months = 12, 12, MOD( MONTH ( date ) + number_of_months, 12 )),   01 ) 

How to use this: the date variable should be the date field you are starting with. You should replace number_of_months with the number of months you want to add to the original date. If the date is 07/01/2017 I want this to go out on 08/01/2017, I’d set number_of_months to 1. If 09/01/2017, I’d set it to 2, etc.

Note: This only works for the first of each month. If you need it to work on any day of the month, use this more complicated solution to account for months having different lengths.

Automating MySQL Backups with Gulp

As I mentioned a few days ago, I’m using Gulp on a new WordPress project. I like to back up my work every night, and since a lot of WordPress config and customization happens in the WordPress editor and widgets, that means backing up the mysql database as well as the code.

Why not use this newfound tool? Let’s do it.

I did some searching and found Gulp WordPress Backup, but it was overkill for what I wanted. But I saw that it used an npm package named mysqldump, for the export, so I grabbed that and started setting up a new task in gulpfile.js:

// add mysqldump as a dependency var mysqlDump = require('mysqldump');  // dumpDatabase gulp.task('dumpDatabase', () => { return new Promise((resolve, reject) => { mysqlDump({ host: 'localhost', user: 'user', password: 'pass', database: 'wp_database', dest: 'backup.sql' }, (err) => { if (err !== null) return reject(err); }); }) .catch((err) => { console.log(err); }); });

Next step: Defining the filename. I just wanted to use today’s date because I intend on running this at the end of each work day. Since gulp is all javascript, this is easy:

var today = new Date(), dd = today.getDate(), mm = today.getMonth()+1 //January is 0! yyyy = today.getFullYear(); if(dd<10) { dd = '0'+dd } if(mm<10) { mm = '0'+mm } today = mm + '-' + dd + '-' + yyyy;

Add this to the gulp task and you are good to go!

gulp.task('dumpDatabase', () => { var today = new Date(), dd = today.getDate(), mm = today.getMonth()+1 //January is 0! yyyy = today.getFullYear(); if(dd<10) { dd = '0'+dd } if(mm<10) { mm = '0'+mm } today = mm + '-' + dd + '-' + yyyy;      return new Promise((resolve, reject) => {         mysqlDump({             host: 'localhost',             user: 'user',             password: 'pass',             database: 'wp_database',             dest: 'SQLBackups/' + today + '.sql' // Outputs to the folder named SQLBackups and uses today's date as the filename.         }, (err) => {             if (err !== null) return reject(err);         });     })     .catch((err) => {         console.log(err);     });  });

Make sure you add mysqldump to your project’s package.json, or at least run npm install mysqldump before using!

Fixing s3_website Java 9 Error with jEnv

When I updated to macOS High Sierra, a bunch of necessary stuff broke: Jekyll, Homebrew, Node.js, and a bunch of gems. s3_website, the tool I use to deploy my Jekyll site to S3, was one of the gems that just completely disappeared. When I went to reinstall it, I got an error that I didn’t have Java installed. Against my better judgment, I went to the URL listed and installed it. Then I ran s3_website push.

After about 30 seconds, I got an error saying that s3_website doesn’t work with Java 9, which was the most recent version at the link. And also the version you get with brew cask install java. Well, shit.

Exception in thread "main" java.lang.ExceptionInInitializerError 	at org.jruby.Ruby.newInstance(Ruby.java:266) 	at s3.website.Ruby$.rubyRuntime$lzycompute(Ruby.scala:4) 	at s3.website.Ruby$.rubyRuntime(Ruby.scala:4) 	at s3.website.model.Config$$anonfun$15.apply(Config.scala:229) 	at s3.website.model.Config$$anonfun$15.apply(Config.scala:227) 	at scala.util.Try$.apply(Try.scala:192) 	at s3.website.model.Config$.erbEval(Config.scala:227) 	at s3.website.model.Site$$anonfun$2.apply(Site.scala:28) 	at s3.website.model.Site$$anonfun$2.apply(Site.scala:27) 	at scala.util.Success.flatMap(Try.scala:231) 	at s3.website.model.Site$.parseConfig(Site.scala:27) 	at s3.website.model.Site$.loadSite(Site.scala:100) 	at s3.website.Push$.push(Push.scala:62) 	at s3.website.Push$.main(Push.scala:40) 	at s3.website.Push.main(Push.scala) Caused by: java.lang.RuntimeException: unsupported Java version: 9 	at org.jruby.RubyInstanceConfig.initGlobalJavaVersion(RubyInstanceConfig.java:1878) 	at org.jruby.RubyInstanceConfig.(RubyInstanceConfig.java:1585) 	... 15 more

After lots of searching, I came across a kind soul on Github suggesting that we use jEnv to define which java environment to use in the directory.

When I first installed jenv, I couldn’t add versions to the tool. I kept getting this error:

ln: /usr/local/opt/jenv/versions/oracle64-9.0.1: No such file or directory

The fix is described here: https://github.com/gcuisinier/jenv/wiki/Trouble-Shooting. Once I added version 8 as well, I switched to version 8 locally with this:

jenv local oracle64-1.8.0.151

Then I opened a fresh Terminal window and ran s3_website again and everything pushed up to s3 without an issue.