Half-Elf on Tech

Thoughts From a Professional Lesbian

Tag: markdown

  • JSON to Markdown

    JSON to Markdown

    Now that I understand how Hugo and JSON work separately and together, it’s time to figure out the next question.

    Can I generate Markdown files based on the contents of a JSON?

    My JSON File

    I started with an SQL DB where my photos have lived for a long time. I exported the albums table to a JSON file (yes, phpMyAdmin can do that), cleaned it up by removing the cruft I didn’t need, and ended up with something like this:

    [
 {
    "categories": "Adventures",
    "tags": "Index",
    "folder": "foobar",
    "title": "Foobar and Baz",
    "desc": "The adventures of Foobar and Baz.",
    "date": "2006-03-24 19:46:40"
  },
  {
    "categories": "Vacations",
    "tags": "Index",
    "folder": "botbob",
    "title": "Bot and Bob",
    "desc": "Bot and Bob's vacation photos",
    "date": "2006-03-24 11:51:04"
  }
]

    By the way, I’m not going to address how to actually make the gallery with images here. Yet. The eventual idea I have here is to run this with WordPress doing that, but since I was messing around with this anyway, I figured I’d reuse what I had.

    The end goal here, remember, is to export the JSON file from WordPress (list all top level categories, let’s say) and have Hugo generate a page for each one. That’s all. But WordPress has so much data, I decided to start with the smaller, simpler stuff.

    Let’s move on.

    Create a Markdown Page from JSON

    Thankfully someone else has already tossed this idea around. I started with reading Dynamic Pages with Hugo

    My idea is: Import any JSON or CSV from any local file or URL and make the JSON or CSV content available in a shortcode or directly in the layout files.

    That was very close to what I wanted, but it would result in me having to make a page for each item. I don’t want that. I want a script to make the .md file for me.

    So I stepped back and stopped asking “How do I make a Hugo page from JSON?” and I asked “How do I make a separate file for each entry in a .json file?”

    Sidebar: My wife was trying to convert the Hebrew year (3595) into all the possible other calendars and asked me to help her with the math. I asked her “What Julian year is that?” She said it BCE so it didn’t work like that, but that was 167. I Googled “What is 167 BCE in the Chinese calendar?” Boom. WikiPedia’s page for 167 BC lists them all. As I pointed out, the trick to these things is not converting it all yourself, but getting from ‘weird’ to ‘Base 10.’ Once you’re at the default (the lowest common denominator), it’s allot easier to figure out other people’s weird.

    This is related to my task at hand. Weird is being specific. Weird is caring what system I’m using. The root of my issue is this: I want to make a Markdown file for every entry in JSON.

    I use Mustache. And, yes, someone already had this idea. He was using mustache-render in Grunt, which I’m already familiar with, but I made a few changes.

    1. I changed the templates folder to my Hugo Archetypes folder. This is my source, my style.
    2. I changed the data folder to my Hugo Data folder.
    3. The output went to (you got this one, right?) the Hugo Content folder.

    This should work, right?

    Mustache Render

    My Gruntfile has this:

        mustache_render: {
      all: {
        options: {
          directory: "partials"
        },
        files: [
          {
            data: "../data/parents.json",
            template: "../archetypes/gallery.mustache",
            dest: "../content/gallery.md"
          }
        ]
      }

    And my gallery.md file is like this:

    ---
title: {{title}}
author: Me
layout: default
permalink: /{{folder}}/
categories: {{categories}}
tags: {{tags}}
---

{{title}}

    Except that doesn’t work. If I changed {{title}} to {{0.title}} then it showed Foo and Bar’s adventure information. Obviously the render was only going to show each entry individually. Now. This could work in a variety of situations, just not mine. Alternatively, I can wrap the output in this: {{#0}} ... {{/0}}

    But what I want is ‘for each key in json, make a file…’ and since I can put the JSON output right into the gruntfile (data: { greeting: "Hello", target: "world" }) then in theory I want to generate this:

    files: [
  {data: OUTPUT FROM KEY ,
   dest: KEY ID,
]

    Okay so how do I tell Grunt to read from a JSON and for each key, do a thing? First I use grunt.file.readJSON:

    pkg: grunt.file.readJSON('../data/parents.json'),

    mustache_render: {
        [...]
            files: [{
                template: '../archetypes/gallery.mustache',
                data: '<%= pkg.1 %>',
                dest: '../content/<%= pkg.1.id %>-foo.md'
            }]
      }
    }

    And this will output item (Bob’s vacations). So stage one is complete. Now I want the 1 in pkg.1 to be iterative or I want to pass a variable to the mustache file and change {{#0}} ... {{/0}} based on its value. Obviously if I had one JSON file for each ‘post’ I wanted to make, this would be easy.

    Multiple Markdown Files from One JSON

    I have no idea.

    Seriously. I spent a few hours on this, lying in bed trying to feel less cold-and-flu-ish. At this point, I don’t really care what I use to do it, but the bones is this: Take one JSON file and split each object into it’s own file.

    Can’t figure that out yet. But hey, there’s my 2016 project!

    FYI: If you reply with ‘try project X!’ without a code example of how to do it, I’ll probably delete your comment. I’ve seen ‘Try handlebars!’ or ‘Try Assemble’ but my issue here is no one has an example of how to do it, and I work best by example. I need that Hello World.

    I am looking at Handlebars and Assemble, by the way, but at 900+ words, it’s time to call a pause on the adventure.

  • Jekyll Table of Contents

    Jekyll Table of Contents

    One of the cool things about MediaWiki, about the only thing I missed, was the built in table of contents display. With Jekyll, since this was a static sort of site, you would have to tell it ‘When you build this page, include a table of contents.’

    And I found a brilliant Jekyll plugin, Jekyll ToC Generator, that added in a beautiful jquery based table of contents with a click back to top feature. But there was a problem. When I installed it and ran a build to test it, my Jekyll site took nine times as long to build.

    In order to generate a Jekyll site, you run the command jekyll build and, on Jekyll 3.0, that gives you the output “done in 8.493 seconds” or so on. Now, if I do a full build on a site I’ve cleaned (there’s a clean command you can run to scrub the site and ensure you get a clean rebuild), it generally takes about a minute. That’s to build a thousand files with a lot of weird trickery. If I’m just rebuilding the changed files, it takes about 10 seconds. Much more reasonable.

    With the ToC Plugin, it took 98 to 100 seconds, every single time. Right away, I knew why. I had included a plugin that had to check every single page on the site on that rebuild, see if it needed a table of contents, and then build the page. Of course that took a long time!

    I’m always talking about needs and wants when I work on websites. It’s a basic principle my high school drilled into me. Understand what you need and how it’s different from your wants. Don’t compromise on needs. Well, I knew that I didn’t need a table of contents, not on every page, so it clearly had to be a ‘mostly want.’

    By contrast, having my site build quickly was a need. A fast build ensured less overhead, less weight, and less time spent. Time is a massive factor in websites. The rendered site has to be fast for users, this much is obvious to everyone. But having your build be fast means you, the site maintainer, spends less time on the parts that don’t make the site better, freeing you to develop and write.

    Also speed is something Jekyll wants to work on. The build different between the 2.3 version of Jeykll and the 3.0-pre beta, is incredible. In 2013, a site with “362 articles with 660 words in average” took around 10 minutes for a full build. I have double the articles, about the same amount of words, and it’s a minute for a full build. It’s faster on my faster laptop (duh).

    The decision tree for Jekyll is more obvious on build than the same one is for WordPress (or MediaWiki) on render. The basic concept is simple. The more complex your site, the longer it takes to generate pages. For WordPress (or MediaWiki, or Drupal, or Joomla, etc etc), the render happens when someone visits your site. For Jekyll and other static site generators, the render occurs when you build the site. That means with Jekyll I can see right away, before I get close to deployment, which means I make the decisions well before the ‘stage’ step of my deployment process.

    What’s more important? The complexities that make your site personal or a fast build?

    Here’s an example from a Jekyll discussion on the matter. Someone had the following code in the templates, which made the date output rather pretty:

    <span>{% assign d = page.date | date: "%d" | plus:'0' %}
    {{ page.date | date: "%B" }} 
    {% case d %}
        {% when 1 or 21 or 31 %}{{ d }}st
        {% when 2 or 22 %}{{ d }}nd
        {% when 3 or 23 %}{{ d }}rd
        {% else %}{{ d }}th
    {% endcase %}, 
    {{ page.date | date: "%Y" }}
</span>

    But that had to run on page builds, which naturally was going to make a site slower. One of the ways Jekyll has improve this was to introduce incremental updates. Only update the pages that need updating. That is a big “baboom!” moment and it let me run the plugin jekyll-last-modified-at (which spits out a last modified date on pages) without any performance hit except on the clean build. Since that only gets called when a page is built, and a page is only built when it changes, it’s a massive improvement for me.

    What does all this have to do with the table of contents?

    Once I pushed it into a ‘want’ and not a ‘need’ I opened my mind to other possibilities. I stopped looking for a Jekyll or Ruby or Liquid based table of contents, and I asked myself “Can Markdown make a table of contents?”

    Markdown is a ‘language’ like HTML that is actually faster to write in than raw HTML but can be read, rendered, and output as HTML. I’m a big fan of HTML and part of why I picked WordPress back in the beginning was that I stumbled on a post where Matt Mullenweg talked about how he didn’t like bbCode and didn’t want it in WordPress. HTML was something we knew. Why make people learn something new?

    It wasn’t until I started blogging more on my iPad and phone that Markdown made sense. Now I’m quite the fan. But I knew that I didn’t know a whole lot about Markdown. I did know that Jekyll used a flavor called ‘kramdown’ (all lowercase) so I read up on that and found that kramdown has a built in built in table of contents generator that was incredibly easy to implement.

    * TOC will be output here
{:toc}

    It’s not something I want (or need) on every page, so I just put that on a few pages. No real overhead added and it’s easy enough to style with CSS. Suddenly I have my cake and I can eat it too.