Half-Elf on Tech

Thoughts From a Professional Lesbian

Category: How To

  • WP JSON API Challenges

    WP JSON API Challenges

    In my frustration of my static process, I ripped everything out by it’s roots and decided to take things in a strict order.

    Decision: All-In-One or Separate?

    The first question was do I want to download everything as one big JSON file and split it out later, or do I want to, from the start, download each post as it’s own file? The time to download shouldn’t change, and if I do it right, I could script it to check if the date stamp on my downloaded file was older than the post date in WP. That means I could have this logic:

    if ( ! post file exists || post file date < post datestamp ) {
        download JSON for this post
    }
    

    That made it an easy win. Let’s split away from the original plan. Now my goal is to extract all the posts from my WP site in JSON format.

    Remember the goal here is to run a static website with WP. There are plugins that convert WP to static HTML, which you can then use to power a site, and this isn’t a bad idea.

    How To: Get all the posts from a WP site via the JSON API?

    This sounded easy. http://local.wordpress.dev//wp-json/wp/v2/posts listed posts, and the v1 comparison chart says that you can list all posts, it stands to reason that I can easily get all my posts.

    In a way, this is a WordPress problem. WP doesn’t want you to show all posts. It would make your site very slow if you had, say, 10,000 posts. Which means that the claim that the JSON API can lists all posts is true, it just doesn’t do them all at once.

    I happen to have 40 published posts on my test site, so when I GET with this URL, it loads all of them: http://local.wordpress.dev//wp-json/wp/v2/posts?per_page=100

    But that isn’t sustainable. And the old trick of making per_page equal to -1 didn’t work.

    I did determine that the JSON API actually knows what the amount of posts is!

    JSON reports X-WP-Total which, in my case is 40

    The header “X-WP-Total: 40” is what I needed. Of course, that is rather weird to get. It means to get all the files, I have to write ‘something’ that does this:

    1. Get http://local.wordpress.dev//wp-json/wp/v2/posts and parse it to get the value of X-WP-Total
    2. Get http://local.wordpress.dev//wp-json/wp/v2/posts?per_page=X-WP-Total

    Okay. So how?

    Well remember how I mentioned a plugin that made a static site already? I was talking about things like Really Static. Save a post, it saves the JSON file! Why not write this plugin?

    Why not use “Really Static”? Well I don’t want to have WP doing more than be my content generator. I want to separate content (the blog) from WP. And that means I will end up writing a plugin…

    Am I In Too Deep? Or Am I Over My Head?

    So far, this has been an incredible amount of work. It’s been a lot of two steps forward and three steps back, over and over and over. And yes, it’s been very frustrating. I’ve given up many times and, if my posts about multiple approaches are any indication, I’ve deleted and restarted many, many times.

    And I have to be honest here… I’m giving up on this overly ambitious plan right now.

    This marks two major failures (three if I could how insane the entire ‘use Jekyll/Hugo to run a gallery’ was). First, I’ve failed to sensibly convert JSON files to MD in a scriptable and reproducible way. Second, I’ve failed to export JSON from WordPress in a simple way. Both those failures can be worked around. I can build a plugin that exports the JSON. It’s not trivial, but having done similar things I’m sure I can do it. The issue is ‘is it worth it’? And right now I think the answer is no.

    I’m back to the drawing board right now, sketching up new plans and ideas. I want to eliminate as many steps as possible. I’d like to leave WordPress alone, let it be WordPress, and similarly let my static generator be static. These failures have taught me more about the interdependencies than I expected, and I’ve learned more from them than I may have if I’d had outright success from the start.

    While I’m starting over, again, I feel hopeful and less ‘I hate you, JSON’ than I thought I would. I now understand things more clearly and I see how I made things far, far more complicated for myself.

    The future of this project is, weirdly, bright.

  • Mailbag: Can I Track Issues with Github Like WP Support?

    Mailbag: Can I Track Issues with Github Like WP Support?

    How do I see all my Github issues for all my projects?

    I was asked this at a WordCamp by someone who had seen my ‘how to get into Github’ series with Carrie Dils.

    If you’re tracking the WordPress forums, https://wordpress.org/support/view/plugin-committer/YOURID will list all of the support requests and reviews for any plugin you have commit access.

    Not a comitter, just someone listed as an author? Use https://wordpress.org/support/view/plugin-contributor/YOURID

    But Github? Turns out it’s super easy!

    https://github.com/search?q=is%3Aissue+is%3Aopen+user%3AYOURNAME&type=Issues

    Basically you can go to the Github Search Page and put inis:issue is:open user:ott42 and you’ll see them all! There’s no RSS feed mind you but you do get emails for all those so you should be okay.

  • Not Ditching ZenPhoto After All

    Not Ditching ZenPhoto After All

    When we last left our heros… We had taken a SQL database, converted it to JSON, split it into 885 separate JSON files, renamed and moved them based on the folder path previously assigned.

    That took a while, but now we’re ready for the fun!

    Convert them to MD

    I already know how to do this one. Run grunt-mustache-render!

    One problem. It didn’t like recursive folders. Pause for laughter. Thankfully I had done a cp and not an mv on my files with Mike’s script, so I just changed that to handle moving .md files instead of .json and it all went fine.

    Turn on Hugo

    I had a theme already so I copied it over and then started making my changes. I needed the layout to be a little different.

    First I needed a list of all the sections and files and I needed to grab all the items that were an ‘Index’ file and format them specially:

    <div id="albums">
    {{ range $section, $taxonomy := .Site.Sections }}{{ range $taxonomy.Pages }}{{if eq .Type "index" }}
    		<div class="indexalbum">
    			<div class="thumb"><a href="{{ .Permalink}}" title="View album: {{ replace $section "-" " " | title }}">IMAGE HERE</a></div>
    			<div class="albumdesc"><h3><a href="{{ .Permalink}}" title="View album: {{ replace $section "-" " " | title }}">{{ replace $section "-" " " | title }}</a></h3>
                {{ with .Params.desc }} {{ . | safeHTML}} {{ end }}</div>
    			<p style="clear: both; "></p>
    		</div>
    {{ end }}{{ end }}{{ end }}
    </div>
    

    I felt very pleased with myself at this point. Obviously I would need to replace “IMAGE HERE” with an image, but I had the basic idea down.

    Inside each post, there was a shortcode mentioned before, {{< gallery >}}, and this is where my drama started. What I wanted I thought would be (fairly) easy. Get a list of all the folders where this index.md file was and list them similar to what I had done before. They were sub-sections (in Hugo Terms), but as it happens, Hugo doesn’t have a way to handle those! There are no templates for subsections. Now this in and of itself wasn’t a deal breaker. I could made do with that shortcode except…

    Limitations

    The thing I really do like about ZenPhoto is that I can drop my new media in a folder and magically knows what’s up. Add a new folder? It knows it’s there. With Hugo (or Jekyll) I found it was incredible hard to do something as ‘simple’ as getting a list of all the items in a folder. With Jekyll I’d need a plugin (which I am certainly not adverse to). Hugo has a readDir call, but it’s limited to the working directory and I wanted to read the images in a CDN folder (on the same server, but still).

    Perhaps ironically, it’s easier to run a gallery with Hugo and Jekyll if you host the images off your own domain. It goes against my goal of having a self-hosted site, though. But for now, it’s not possibly to do what I wanted.

    Do I feel like I wasted time?

    No! In fact, I’ve learned a lot of things that tell me where the foibles are in my plans.

    ustwo actually did make a React powered site and in looking at their source code I see they did it by making the site ‘all one page.’ This is something I can understand today and wrap my head around. Because the alternative would be something like this:

    1. Write a post
    2. Run a script to grab all the pages via the JSON API as individual .json files
    3. Run a script to convert them into .md
    4. Build static site (test and then deploy)

    It’s not actually that horrible when I put it that way. It means everyone could write on WordPress and, when they’re ready to deploy, I just run my script, push to my dev site, validate, and push to live.

    But. That loses a lot of what makes WordPress cool. Editing posts on the fly, pushing live updates, and scheduling posts all becomes a nightmare. For my Gallery idea, it means that WordPress remains a viable option except for the fact that I can’t easily move from ZenPhoto to WordPress and WordPress behaves like a blog when I don’t want it to be one.

    I’m all about using WordPress for things other than blogging but the sad truth is it’s still structured like a blog. Custom Post Types are doing a great job, and a lot of what I do on my estore site is similar to how I’d want to handle a gallery. I could do posts as posts and organize them as categories, for example, and structure the whole theme to handle that…

    Or I could just stay on ZenPhoto, which does all that out of the box.

    And yes. That’s what I’m doing.

  • Ditching ZenPhoto for Hugo?

    Ditching ZenPhoto for Hugo?

    This started out as a 230 word bullet point list of how to do things. It’s now two entire posts!

    (Almost) Leaving ZenPhoto

    I have to preface this by telling you that I didn’t in the end. Yeah. I know. I planned to. And here’s why:

    I like ZenPhoto a lot. It’s a great Gallery tool, and after Gallery2 went to Gallery3 and I hated it, I was quite in love with them. But. They’ve started to inch towards being a general CMS and the whole ZenPhoto/ZenPhoto20 split left a weird taste in my mouth.

    Plus the DB I have is already 65 megs, which has started to behave a little oddly. Now I have 38,534 images in 885 albums, and to a degree that makes sense. If you think about it in WordPress terms, I have 885 taxonomies (tags and categories), and 38,534 posts. Given that each post in this example would be a sentence at most (describing the image), the WordPress DB would be around 15megs based on some napkin math I did. Validating this, there’s a WordPress Joke DB with 40k Jokes and it’s only 14 megs.

    Oh and did I mention that ZenPhoto has no export feature? I guess there’s a reason the first PHP script I wrote was a simple Gallery. All I really want is to display images in a structured format.

    But. As I said before, I did not leave ZenPhoto! Why not? Because I was unable to do what I wanted to in pure Hugo, and because porting everything over to WordPress is incredibly hard and complex.

    Why Not WordPress?

    WordPress has a lot of awesome advantages. It auto generates thumbnails for one, which Hugo does not, so porting the gallery over would save me one level of complication. But I have 38,534 images to import. No matter how you slice that, I’d have to import them and associate them with the right post. Probably manually.

    With a static generator, I can very easily assign a variable for each album to say ‘get images from this folder and display them.’ And if there are folders, list each folder as a link to another page. By contrast, I would have to use a plugin like NextGen gallery to do this in WordPress, and while I do like the plugin, it’s got the same issue that ZenPhoto has for me. It’s doing too much. And, to be honest, part of this process is to limit my potential security issues. Adding in really big, really complex plugins is the opposite of that, no matter how rigorously reviewed they are.

    Besides, if I hated it, I knew I could import MD files into WordPress. Someone already did it with Jekyll after all.

    Get and Grep the Data

    No matter what I needed the data. Thankfully I knew how to do this with phpMyAdmin already. I dropped the _albums table in JSON format. Why JSON? Well as I’d already learned, I could generate a Markdown file from a JSON pretty easily. Of course, I got stumped on having it loop through the JSON file, but I have a plan for this.

    On export, albums.json was 643 KB while images.json was 61.4 MB. And they were all one line. I’m not using images.json just yet.

    I tossed it into BBedit and scrubbed the data, removing everything I didn’t need. This did not change the file size at first, since I also made the JSON human readable. There was a lot of gripping here, but I took each entry from 16-22 items down to six: tags, categories, folder path, title, description, date. And the file went down to 143 KB. One-sixth, more or less.

    Finally I had to split each entry because, as I’d learned, trying to loop through a nested JSON array in grunt is complicated. So fine, I’ll do it manually. But how? Answer: By loving linux. You can split files!

    $ split -a 6 -l 8 ../albums.json album-
    

    There’s also csplit, which is contextual split. Either way, I now have 885 album-xxx.json files. Whew.

    Rename and Move the Files

    I messed around with linux commands to move files based on their folder paths and rename them based on the last variable. I dreaded having to just move them and then going to mess with the renames. So after two days, I asked Twitter and was saved by Mike Little and Kailey Lampert. Between them I learned that there was a jq tool for messing with JSON (homebrew install it) and that a shell script is a godsend.

    #! /bin/sh
    
    # awk script:
    # set field seperator to :
    # strip comma from field 2
    # strip quotes from field 2
    # strip lead space from field 2
    # strip trail space from field 2
    # print field 2
    
    for FILE in `ls *.json` ; do
        NAME=`cat $FILE | grep folder | awk 'BEGIN { FS= ":" } { 
            gsub( ",", "", $2 );   \
            gsub( "\"", "", $2 );  \
            gsub(/^[ \t]+/,"",$2); \
            gsub(/[ \t]+$/,"",$2); \
            print $2 
        }'`
        
        mkdir -p $NAME;
        cp "$FILE" "$NAME/index.json"
    done
    

    I ended up changing it to reduce future broken URLs. But basically everything except the mkdir comes from Mike. He’s my hero.

    To Be Continued

    Once I had all the files, it was time to change them all to Markdown files and whip out my templates!

    But you’ll need to wait for that because this post got really long.

  • JSON to Markdown

    JSON to Markdown

    Now that I understand how Hugo and JSON work separately and together, it’s time to figure out the next question.

    Can I generate Markdown files based on the contents of a JSON?

    My JSON File

    I started with an SQL DB where my photos have lived for a long time. I exported the albums table to a JSON file (yes, phpMyAdmin can do that), cleaned it up by removing the cruft I didn’t need, and ended up with something like this:

    [
     {
        "categories": "Adventures",
        "tags": "Index",
        "folder": "foobar",
        "title": "Foobar and Baz",
        "desc": "The adventures of Foobar and Baz.",
        "date": "2006-03-24 19:46:40"
      },
      {
        "categories": "Vacations",
        "tags": "Index",
        "folder": "botbob",
        "title": "Bot and Bob",
        "desc": "Bot and Bob's vacation photos",
        "date": "2006-03-24 11:51:04"
      }
    ]
    

    By the way, I’m not going to address how to actually make the gallery with images here. Yet. The eventual idea I have here is to run this with WordPress doing that, but since I was messing around with this anyway, I figured I’d reuse what I had.

    The end goal here, remember, is to export the JSON file from WordPress (list all top level categories, let’s say) and have Hugo generate a page for each one. That’s all. But WordPress has so much data, I decided to start with the smaller, simpler stuff.

    Let’s move on.

    Create a Markdown Page from JSON

    Thankfully someone else has already tossed this idea around. I started with reading Dynamic Pages with Hugo

    My idea is: Import any JSON or CSV from any local file or URL and make the JSON or CSV content available in a shortcode or directly in the layout files.

    That was very close to what I wanted, but it would result in me having to make a page for each item. I don’t want that. I want a script to make the .md file for me.

    So I stepped back and stopped asking “How do I make a Hugo page from JSON?” and I asked “How do I make a separate file for each entry in a .json file?”

    Sidebar: My wife was trying to convert the Hebrew year (3595) into all the possible other calendars and asked me to help her with the math. I asked her “What Julian year is that?” She said it BCE so it didn’t work like that, but that was 167. I Googled “What is 167 BCE in the Chinese calendar?” Boom. WikiPedia’s page for 167 BC lists them all. As I pointed out, the trick to these things is not converting it all yourself, but getting from ‘weird’ to ‘Base 10.’ Once you’re at the default (the lowest common denominator), it’s allot easier to figure out other people’s weird.

    This is related to my task at hand. Weird is being specific. Weird is caring what system I’m using. The root of my issue is this: I want to make a Markdown file for every entry in JSON.

    I use Mustache. And, yes, someone already had this idea. He was using mustache-render in Grunt, which I’m already familiar with, but I made a few changes.

    1. I changed the templates folder to my Hugo Archetypes folder. This is my source, my style.
    2. I changed the data folder to my Hugo Data folder.
    3. The output went to (you got this one, right?) the Hugo Content folder.

    This should work, right?

    Mustache Render

    My Gruntfile has this:

        mustache_render: {
          all: {
            options: {
              directory: "partials"
            },
            files: [
              {
                data: "../data/parents.json",
                template: "../archetypes/gallery.mustache",
                dest: "../content/gallery.md"
              }
            ]
          }
    

    And my gallery.md file is like this:

    ---
    title: {{title}}
    author: Me
    layout: default
    permalink: /{{folder}}/
    categories: {{categories}}
    tags: {{tags}}
    ---
    
    {{title}}
    

    Except that doesn’t work. If I changed {{title}} to {{0.title}} then it showed Foo and Bar’s adventure information. Obviously the render was only going to show each entry individually. Now. This could work in a variety of situations, just not mine. Alternatively, I can wrap the output in this: {{#0}} ... {{/0}}

    But what I want is ‘for each key in json, make a file…’ and since I can put the JSON output right into the gruntfile (data: { greeting: "Hello", target: "world" }) then in theory I want to generate this:

    files: [
      {data: OUTPUT FROM KEY ,
       dest: KEY ID,
    ]
    

    Okay so how do I tell Grunt to read from a JSON and for each key, do a thing? First I use grunt.file.readJSON:

    pkg: grunt.file.readJSON('../data/parents.json'),
    
        mustache_render: {
            [...]
                files: [{
                    template: '../archetypes/gallery.mustache',
                    data: '<%= pkg.1 %>',
                    dest: '../content/<%= pkg.1.id %>-foo.md'
                }]
          }
        }
    

    And this will output item (Bob’s vacations). So stage one is complete. Now I want the 1 in pkg.1 to be iterative or I want to pass a variable to the mustache file and change {{#0}} ... {{/0}} based on its value. Obviously if I had one JSON file for each ‘post’ I wanted to make, this would be easy.

    Multiple Markdown Files from One JSON

    I have no idea.

    Seriously. I spent a few hours on this, lying in bed trying to feel less cold-and-flu-ish. At this point, I don’t really care what I use to do it, but the bones is this: Take one JSON file and split each object into it’s own file.

    Can’t figure that out yet. But hey, there’s my 2016 project!

    FYI: If you reply with ‘try project X!’ without a code example of how to do it, I’ll probably delete your comment. I’ve seen ‘Try handlebars!’ or ‘Try Assemble’ but my issue here is no one has an example of how to do it, and I work best by example. I need that Hello World.

    I am looking at Handlebars and Assemble, by the way, but at 900+ words, it’s time to call a pause on the adventure.

  • Using JSON with Hugo

    Using JSON with Hugo

    I know I’ve been talking about Hugo a lot, but the whole reason I say and wrapped my head around it was that I wanted to make a website using JSON.

    Making something out of the JSON API in WordPress isn’t easy, and I decided to start with something hard. I thought I would make a Multisite network to be the back end of my site, and then call the JSON API to generate the content.

    Why?

    It doesn’t make my site (much) faster, though it makes it more cacheable. Mostly what it does is lets me separate church and state. I can use WordPress to write content, and then call it however I want. That would be Hugo. If I build the site via Hugo locally, then every time I generate the files it can pull in the data on the fly.

    Enable the API

    This is easy. Install WordPress 4.4 and install v2 of the Rest API plugin. I also installed Jetpack for reasons of using Calypso.

    You absolutely have to use the Rest API plugin. WordPress 4.4 adds in some of the bits, but the API plugin gives you the actual endpoints. You’ll need these later.

    Call the API

    This was also easy. Kind of. Making a jquery file that called the content was fairly straightforward but… Making it responsive and reactive was hard.

    What I really wanted to do was have some web app, js powered without node, that just called the data and kept the url structure. And for that what I need is a JSON client.

    It was incredibly hard to find how to make a static HTML website powered by JSON. I’d see Rachel Baker’s presentation on making a WordPress theme with the JSON API but that wasn’t what I wanted to do. I mean, in a way it was. The theme itself was what I wanted, but I didn’t want to run it with WordPress.

    I did a lot of research and finally stumbled onto restful.js by marmelab. I read their blog post on the product and it looked like what I wanted. I needed something that would run without Node.js since I wanted to do this on Apache.

    Consuming a RESTful Web Service Dynamically

    And this is when I starting banging my head against a table. None of the solutions I’d found were working for me. Nothing. Rachel was right. I was in a bit over my head. Oh she was nice enough not to say it that way, but they way she looked at me was very Jewish Mom of her.

    In order to stop feeling useless, I decided to learn Hugo. It had come up a few times on my search for a dynamic static site. Hugo has LiveReload which, if you’re familiar with most static generators, is pretty cool. The idea is the pages rebuild as you edit the content (or theme) files.

    If you run hugo server locally on your laptop, you can see this today. Hugo upped the ante by tossing in the ability to run it on the server and the ability to watch the data files. It also has the ability to call JSON remotely.

    I feel it’s important to note that you cannot trigger that LiveReload when you’re using external URLs. But anytime you edit a file and LiveReload is triggered, Hugo will read the URL content from your cache. If you’ve disabled cache, it will download it fresh. Don’t disable cache unless you have an unlimited API. And no, I don’t know how long the cache is good for.

    Sadly, OAuth or other authentication methods are not working yet, but for now that’s okay because I’d been in over my head for so long, I stepped back.

    Understanding JSON First

    Okay fine. Once I realized I’d tried to do too much at first, I went back to that static filmography I had.

      {
      	"role": "actor",
        "title": "White Christmas",
        "slug": "white-christmas",
        "type": "Movie",
        "character": "Phil Davis",
        "notes": null,
        "dates": 1954
      }
    

    This is basic stuff. A simple chunk by chunk. And to call the file in a shortcode, I did this:

    {{ range $.Page.Site.Data.filmography }}
    	{{ if eq $role .role }}
    		<li>{{ .title }}: <strong>{{ .character }}</strong> </li>
    	{{ end }}
    {{ end }}
    

    Actually I did a lot more but you get the idea. And if you’re wondering about the if statement at the top, it’s so I can pass a parameter to my shortcode:

    {{< filmography role="actor" >}}

    It took me longer than I’d like to figure that out. At first I had multiple filmography files (actor, writer, producer) and I was trying to look through them all. In the end, I realized this simple JSON and a simple call was better.

    A Little More JSON

    There was another case, though. I had lists of episodes compiled into separate JSON files, one per show. This is logical and maintainable after all, but it meant I couldn’t use the same logic loop in the same way.

    What I really needed was to loop through $.Page.Site.Data.episodes.SHOWNAME in order to get the data from that show. And I couldn’t add on a variable to my range call.

    But as it happens, Hugo is clever and if I say {{ range $.Page.Site.Data.episodes }} it will give me the file episodes.json if that exists. But if there happens to be a folder called episodes then it will give me all the JSON files in there.

    I read through the data documentation on Hugo a few times to understand the hierarchy but it breaks down really logically:

    ├── data
    |   ├── episodes
    |   |   ├── continum.json
    |   |   └── fringe.json
    |   |   └── sense8.json
    |   ├── movies.json
    |   ├── musicals.html
    

    Data {dot} episodes {dot} continum

    And if I ran this:

    {{ range $.Page.Site.Data.episodes }}
    	<li>{{ .show }}<li>
    {{ end }}
    

    Then I got this:

    • continum
    • fringe
    • sense8

    I already knew I could do a range within a range, and finally I came up with this:

    {{ $show := $.Page.Params.show }}
    
    {{ range $.Page.Site.Data.episodes }}
    	{{ if eq .show $show }}
    		<li>{{ .title }}</li>
    	{{ end }}
    {{ end}}
    

    The shortcode checks the post it’s on and, if the Front Matter has ‘show’ defined, it will give me a list of all titles (episode titles) for that show.

    I added in an extra check for the season parameter. This one is passed through the shortcode:

    {{< episodelist season="1" >}}

    And that, in turn is called in the shortcode above the output for title:

    {{ range where .episodes "season" "==" $season }}

    So nothing will display unless the show in the JSON file matches the show on the page with the shortcode, and the season matches the one defined in the shortcode.

    Now I actually have a more robust check. If no season is defined, it shows all episodes (useful for one-season shows, right?). But you get the idea.

    How Did This Help?

    Remember my JSON example before? Here’s what it looks like for Fringe:

    {
    "show" : "fringe",
    "episodes": [
      {
        "epnum": "1",
        "season": "1",
        "title": "Pilot",
        "slug": "pilot",
        "airdate": "2008-09-08T22:00:00.000Z",
        "rating": "5",
        "summary": "Walter gets a cow named Gene to experiment on ethically."
      },
      ...
    ]}
    

    Instead of just started with those naked chunks, I have a definition of the show name (fringe – lowercase because I want to compare it to other lowercase things and I’m lazy) and the definition of a section for episodes. That’s the whole reason why that call for seasons works the way I want it to.

    This helped me to understand and visualize JSON. By taking a simple JSON list (the filmography) I learned the basics of connecting code and content. Then by taking the more complex scenario, where I needed nested data, I understood that relationship.

    The next step will be making a simple WordPress site and calling the JSON remotely. Obviously I have to figure out how to trick it into making fake permalinks, but baby steps. Baby steps.