I encountered a strange problem today. Someone could access their website from their cell phone but not their laptop and said a friend a few states away also couldn’t reach it. But I could reach it just fine.
When we started digging into what was going on, I found that he was using Google’s public DNS. When I switched to Google’s DNS, I couldn’t reach the site either.
In the days.animate section, I make sure to prevent the circle from ever reaching 100%. It will get extremely close as the years pass and the numerator comes closer and closer to the denominator, but the denominator will always be slightly larger. Thanks to Eric Davis for helping me come up with the solution.
Javascript Counter with circular progress
The results:
The javascript, which counts the time since a certain date:
I have multiple old Aperture photo libraries that I canât really use anymore. Aperture doesnât run on El Capitan and I donât have any system that it can run on. Iâve been using Adobe Lightroom for the past four years anyway. So I did some research into options into how I can retrieve my photos.
Opening the package
The great thing about Aperture was that it always kept the master images and applied edits on the fly. So I knew that I could get the master images out of Aperture. The way you can get to them is right-clicking on the library file and selecting âShow Package Contents.â Then you can take the masters folder and copy it out of the library.
I’ve been on vacation and spend the last two days catching up and not doing a lot of learning, so I’ve been lazy in putting up TIL posts. That is over. (I did, however, push some updates to my Apple Photos Analysis project.) Here is a small collection of things I learned in the last week.
Amending commits
Say you forgot to add a file to your last commit or you made a typo in your commit message. You can amend it!
Make the necessary changes, then do this:
git commit --amend-m"Commit message here"
If you’ve already pushed it to an external repository, you’ll need to force the push since the external repo will look like it is ahead. If branch protection is turned on, you’ll need to make a new commit. Make sure you aren’t overwriting anything important!
Adding data labels to the top of bar charts in Matplotlib
Matplotlib is a great plotting library for Python.
defautolabel(rects):# attach some text labels forrectinrects:height=rect.get_height()plt.text(rect.get_x()+rect.get_width()/2.,5+height,'%d'%int(height),ha='center',va='bottom')rect=plt.bar(xs,counted_hours,color=color)# To use: autolabel(rect)
Saving images in matplotlib
plt.savefig('directory/filename.png')
Counting items that match a regex pattern
defhour_finder(regex,lines):time_counter=0forlinlines:ifre.match(regex,l):time_counter=time_counter+1returntime_counter# To use hour_finder('^8:[0-9]{2,}:[0-9]{2,}sPM',time_csv)
Splitting!
Splitting by a space ' ' and choose the item after the split ([1] because counting starts at 0)
See it in action below. Styling the buttons will be left as an exercise to the reader:
This is Div-1!!!
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Phasellus tempus, mauris in sollicitudin commodo, eros mauris euismod arcu, eget faucibus enim tellus a turpis. Suspendisse malesuada interdum elit sit amet rutrum. Nam ut elit lobortis, lobortis nisi sed, tincidunt nibh. In consequat eleifend quam, vestibulum efficitur velit scelerisque nec. Quisque quis rutrum mauris, ut ultricies ipsum. Cras interdum ipsum in diam venenatis fermentum. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia Curae; Donec mollis tincidunt est vitae dapibus. Aenean cursus, ipsum ac pellentesque malesuada, mauris magna pretium nunc, eu mollis dolor ipsum sit amet felis. Vestibulum sed erat eu dolor condimentum viverra blandit sit amet mauris. In arcu nisi, sollicitudin a dui a, laoreet pellentesque libero.
Woah, this is Div-2!~!~!
Suspendisse bibendum sem vitae tellus maximus dictum. Nunc vestibulum tellus sem, sit amet efficitur ligula porta sed. Aliquam erat volutpat. Suspendisse in enim elementum, iaculis magna eu, placerat enim. Aliquam auctor ultricies ligula eleifend sodales. Vestibulum at leo leo. Praesent sollicitudin ipsum ut maximus malesuada.
If you copied this example and it isn’t working, make sure you gave the divs and buttons the correct classes and you included jQuery in your header:
-- Getting path to this filetellapplication"Finder"togetfolderof(path tome)as Unicode textsetcurrentLocationtoresultsetcurrentPOSIXtoPOSIXpathofcurrentLocation
Running other AppleScript files
run scriptfile"HD:path:to:file.scpt"-- How it is used with above file pathrun scriptfile(currentLocation&"reformat_with_textwrangler.scpt")
Running multiple shell commands
do shell script"command 1; command 2;"-- How it is used with above file path to change directory and run a scriptdo shell script"cd ""ed formofcurrentPOSIX&"; python "&"count_days.py"
Given that the lists days and count are built above by the Counter(), you can pass them to matplotlib for charting:
frommatplotlibimportpyplotasplt######## Bar Chart ######## xs=[i+0.1fori,_inenumerate(days)]plt.bar(xs,count)plt.ylabel("Number of photos taken")plt.title("Photo frequency by day")plt.xticks([i+0.5fori,_inenumerate(days)],days)plt.savefig('img/weekdays_bar.png')plt.clf()######## Pie chart ######## colors=['red','orange','green','purple','lightcoral','lightskyblue','yellowgreen']explode=[0,0,0,0,0,0.1,0]plt.pie(count,explode=explode,labels=days,colors=colors,autopct='%1.1f%%')plt.axis('equal')plt.suptitle("Percent of total photos taken on a given day of the week",fontsize=18)plt.savefig('img/weekdays_pie.png')
If you donât want to save the images, you could just show them instead with plt.show()
plt.clf() clears the figure so you can plot something else on it. Otherwise youâd need to close it before continuing. plt.close() can do that.
Depending on the source CSV, the above creates these two charts:
I took a 10-day break from my TILs and now I’m reinvigorated and back on track. In the past 10 days I spent a lot of time doing things I already knew how to do, but I also worked on a small project related to data visualization to see if I could apply some things I learned over the past few months. The result was my recent blog post on Steph Curry’s stats.
A new project
I got a lot of good feedback on Facebook on some more things I can explore (and about how little I understand basketball.) I decided to work on another personal project to apply some of the data science I’ve been learning by reading Joel Grus’s Data Science from Scratch. I decided to find some data (photo dates and locations), extract it (AppleScript), format it (grep FTW), analyze it (forthcoming in Excel and Python), then visualize the insights (Python and/or D3). You can follow my progress on GitHub.
Dealing with files in AppleScript
AppleScript natively deals with file paths with colons: Macintosh HD:usr:local:bin: If you get a file path then want to pass it to the shell, you’ll first need to turn it into a POSIX path:
If you want to create a file in the same folder as a script you are running, you might have to jump back and forth between AppleScript paths and POSIX paths because it is easier to make files by using the command do shell script. Notice the use of quoted form of, which quotes the file path and keeps you clear of pesky errors caused by characters in the path that need to be escaped:
-- Creating the file in the same folder as this scriptsetscriptPathtoPOSIXpathof((path tomeas text)&"::")do shell script"> ""ed formofscriptPath&"photo_dates_location.csv"
Then if you want to get the path to the file you just created and make an it an alias to reference later:
Suppose you have a [spreadsheet full of photos, the date they were taken, and the days of the week they were taken on] (https://github.com/cagrimmett/apple_photos_analysis) and you want to count how many were taken on Monday, Tuesday, etc. Then you’d use the function COUNTIFS(range,argument) to work it out.
Example: Suppose I have the days of the week in column A and cell H2 contained the word I was looking for, Monday. Then my formula would be:
=COUNTIFS(A2:A8500,H2)
I quickly repeated this for the cells that contained the values for the other days of the week and got instant results:
For example: here is how I’d combine columns of hours and minutes and put a colon in-between: =[@Hour] & ":" & [@Minute]
Convert hours.minutes to hh:mm:ss in Excel
Take 275.75 and convert it to 275:45 =CELL/24 converts the hours.minutes to days Then you format the column by Custom > Time > 37:30:55 (hh:mm:ss)
SQL Dates, Concatenation, and Grouping
I learned some useful things today in SQL: SUM(), CONCAT_WS() to get CSV output, DATE() to get the date part of a datetime stamp, and GROUP BY to get the sums grouped by another column
There are almost always multiple paths to your end goal. If you are stuck spinning your wheels on one path, try another.
If you get stuck, write down your end goal and then write down the discrete steps you need to get there. Then, at each bullet point, ask, “How can this step be eliminated or completed in a better way?”
Every time I go through this exercise I get some traction and start moving forward again.
If you’ve ever tried to write a sum() function in Excel while you have filters on, you likely didn’t get the result you expected. The sum() function adds up what is in the raw cells. It does not take whether or not they’ve been hidden via a filter into account.
subtotal() is the function you are looking for. Specifically, subtotal(9,range). The 9 refers to sum for hidden (not at a result of filtering, though) and non-hidden rows. See the documentation.subtotal() functions only on the results of filtered data.
=SUBTOTAL(9,G1670:G640501) gave me the sum of the the filtered data from G1670 to G640501. (You can see why filtering would be important with such a large data set!)
Remind yourself why
Sometimes I spend too much time and effort trying to convince myself to do something I don’t want to do. This leads to general angst and despair.
My wife reminded me yesterday that the best way of convincing myself is to reflect on why I’m doing it in the first place.
Whatever your reason why is, as long as it outweighs the cost of not doing it, you’ll get over it and find a way. I keep an index card with some why bullet points on it nearby to keep things in perspective.
According to the Jekyll docs, the jekyll-paginator plugin does not support categories or tags. That means there is no way to exclude a category as a whole. You can only exclude posts individually by including hidden: true in each post’s YAML front-matter.
I’m hiding all of my TIL posts from the front page of my site, so I did a global find and replace to accomplish it.
Mirror of If statement in Liquid
Liquid has a neat concept that is the inverse of an If statement. It executes if a condition is not met.
{%unlesspost.categoriescontains'TIL'%} Code here to display all the things! {%endunless%}
Inserting via mysqli()
Using the mysqli() function I wrote about yesterday, here is how to insert something into a table:
$connect=newmysqli($server,$db_username,$password,$db);$connect->query("INSERT INTO `table_name` (username) VALUES ('$user');");
Make sure your variables are sanitized first! You don’t want someone doing a SQL injection.
Eric Davis told me that mysqli() is the old way of doing things and I should check out PDO instead.
This checks for a table called scorecard_test and creates it if it doesn’t exist. The SQL parameters for the columns are:
An integer called ID that is the primary key and auto increments
A username that can’t be NULL
A column called counter that has a default value of 1 if there is nothing passed, and the length can’t be longer than one digit
A column that holds the current timestamp.
// SQL syntax$sql="CREATE TABLE IF NOT EXISTS scorecard_test ( id int(11) NOT NULL AUTO_INCREMENT PRIMARY KEY, username varchar(255) NOT NULL, counter int(1) NOT NULL DEFAULT 1, time TIMESTAMP )";// Connecting, sending query, showing errors, and closing connectionif($connect->query($sql)===TRUE){echo"Done!";}else{echo"Error: ".$connect->error;}$connect->close();
Philosophies like Agile Development are great for teams working on large projects, but for someone just getting started with development and working on small projects, they can be a little much. For situations like this, Readme Driven Development strikes the right balance of planning out your project, writing documentation, and actually doing the work.
Write your Readme first! (H/t to Eric Davis for telling me about this.)
WordPress Plugin Basics
I have a food & drink website where I post regularly. Lately I’ve been trying lots of kinds of coffee to dial in my ROK Espresso maker and I’m starting to reach the limits of how many I can accurately remember. I’ve been wanting to make a WordPress plugin for a while, so why not make a plugin that makes a Coffee Reviews custom post type? (The main goal here is to write my own plugin and learn how it is done, so please don’t suggest already made plugins that I could use.)
Hooks allow you to tap into WordPress without editing the core files. (Rule #1 is to never edit the core files.) You pass functions through hooks to extend or change WordPress functionality.
The register_activation_hook() runs when you press “Activate” on a plugin. Since I’m making a custom post type, I’ll want to use the activation hook to refresh permalinks once the post type is created. If your plugin needs a custom database table, you can create that with a function passed to this hook.
The register_deactivation_hook() is basically the reverse of the activation hook. When you deactivate the plugin, the items passed to this hook fire. I’ll want to clear the permalinks again.
I made a barebones plugin tonight that registers the custom post type. I definitely have more work to do, but saving, hitting refresh, and seeing this appear never gets old!
{%highlightliquid%} Put your foo here bar end foo {%endhighlight%}
I need to go back though my previous posts and remove the Pygments highlighting then uninstall the plugin. Rouge supports fewer languages than Pygments, but Rouge supports all of the languages I use, plus Liquid and Markdown, which Pygments doesnât.
Downloading WordPress.com Email Subscribers
I learned today that it is possible to download a CSV of the email subscribers to your WordPress.com blog (or WordPress.org blog powered by WordPress.comâs JetpackArchived Link). They donât make it obvious, but the option is there:
When doing development work, it is best if you have a separate, dedicated area for each of these things:
Development – The area where you do your development work.
Staging – The area where you deploy changes and thoroughly test them. The only thing that should be out of sync with the production environment is the single change you are testing.
Production – The live system for stable, tested code.
I’ve attempted to replicate this best-practice with the development of my Toggl slash command for Slack. My development environment is on a Homestead box on my local machine. The staging area is a folder on my dev server that a /toggldev command posts to. Once the changes here have been confirmed, I commit my code to the repository and then deploy it to production. The production environment is a folder on my main server that the /toggl command posts to. This is the one my team uses and it stays up to date with releases in the repository.
Of course, the entire process is better if you have version control:
A method for committing changes in Git
Eric Davis advises me that it is better form to commit changes after each feature you want to release is completed rather than commit after a full day of work. If you don’t commit after each feature (and develop each feature in a separate branch), you’ll end up with a mass of code changes in each commit that is about as tangled as spaghetti.
Right now, since I’m the only one working on my Toggl project and the changes I’m making aren’t major, I’ve only been working directly on the master branch. This Atlassian tutorial on branches has convinced me that I should be using branches on a daily basis for development, so I’m going to start that this weekend with some new features I plan to work on.
For the Toggl Slack slash command, I’m using it to check if someone’s Toggl API key is in the array in variables.php:
if(key_exists($user_name,$toggl_api_key_mapping)==FALSE){echo":warning: You haven't set up your Toggl API key with this command yet. Get it at the bottom of https://toggl.com/app/profile and send it to @$your_admin_name.";}
Notes:
Case sensitive. If you need it to be insensitive, use strtolower() on the $key
Boolean (returns TRUE or FALSE)
Looking up a key and returning a value in an array
$array['key']
For the Toggl Slack slash command, I’m using it to set someone’s API key from the array in variables.php:
If you are serving something on your webserver that you don’t want anyone else to be able to access, you can restrict access to by adding this snippet to your site’s .htaccess file:
log.csv>orderallow,denydenyfromall
If you need only a certain IP address, you can achieve this by adding allow from 0.0.0.0 (replace that number with your IP address):
