Half-Elf on Tech

Thoughts From a Professional Lesbian

Tag: health check

  • Automate Your Site Checks with Cron (and WPCron)

    Automate Your Site Checks with Cron (and WPCron)

    I have a self-hosted healthchecks.io instance (mentioned), and I use it to make sure all the needful cron jobs for my site actually run. I have it installed via Docker, so it’s not super complex to update and that’s how I like it.

    The first cron jobs I monitored were the ones I have setup in my crontab on the server:

    1. Run WP ‘due now’
    2. Set daily random ‘of the day’
    3. Download an iCal file
    4. Run a nightly data validity check

    I used to have these using WP Cron, but it’s a little too erratic for my needs. This is important, remember this for later, it’ll come back up.

    Once I added in those jobs, I got to thinking about the myriad WP Cron jobs that WordPress sets up on its own.

    In fact, I have a lot of them:

    +------------------------------------------------+---------------------+-----------------------+---------------+
    | hook                                           | next_run_gmt        | next_run_relative     | recurrence    |
    +------------------------------------------------+---------------------+-----------------------+---------------+
    | rediscache_discard_metrics                     | 2025-04-25 17:51:15 | now                   | 1 hour        |
    | wp_privacy_delete_old_export_files             | 2025-04-25 18:16:33 | 20 minutes 38 seconds | 1 hour        |
    | wp_update_user_counts                          | 2025-04-25 20:30:03 | 2 hours 34 minutes    | 12 hours      |
    | recovery_mode_clean_expired_keys               | 2025-04-25 22:00:01 | 4 hours 4 minutes     | 1 day         |
    | wp_update_themes                               | 2025-04-26 04:57:57 | 11 hours 2 minutes    | 12 hours      |
    | wp_update_plugins                              | 2025-04-26 04:57:57 | 11 hours 2 minutes    | 12 hours      |
    | wp_version_check                               | 2025-04-26 04:57:57 | 11 hours 2 minutes    | 12 hours      |
    [...]
    +------------------------------------------------+---------------------+-----------------------+---------------+
    

    While I could manually add them all to my tracker, the question comes up with how to add the ping to the end of the command?

    The Code

    I’m not going to break down the code here, it’s far too long and a lot of it is dependant on my specific setup.

    In essence, what you need to do is:

    1. Hook into schedule_event
    2. If the event isn’t recurring, just run it
    3. If it is recurring, see if there’s already a ping check for that event
    4. If there’s no check, add it
    5. Now add the ping to the end of the actual cron even
    6. Run the event

    I actually built out code like that using Laravel recently, for a work related project, so I had the structure already in my head and I was familiar with it. The problem though is WP Cron is nothing like ‘real’ cron.

    Note: If you really want to see the code, the beta code can be found in the LWTV GitHub repository. It has an issue with getting the recurrence, which is why I made this post.

    When CRON isn’t CRON

    From WikiPedia:

    The actions of cron are driven by a crontab (cron table) file, a configuration file that specifies shell commands to run periodically on a given schedule. The crontab files are stored where the lists of jobs and other instructions to the cron daemon are kept. 

    Which means crontab runs on the server time. When the server hits the time, it runs the job. Adding in jobs with the ping URL is quick:

    */10 * * * * /usr/bin/wp cron event run --due-now --path=/home/username/html/ && curl -fsS -m 10 --retry 5 -o /dev/null https://health.ipstenu.com/ping/APIKEY/due-now-every-10

    This job relies on the server being up and available, so it’s a decent metric. It always runs every ten minutes.

    But WP Cron? The ‘next run’ time (GMT) is weirdly more precise, but less reliable. 2025-04-25 17:51:15 doesn’t mean it’ll run at 5:51pm GMT and 15 seconds. It means that the next time after that timestamp, it will attempt to run the command.

    Since I have a scheduled ‘due now’ caller every ten minutes, if no one visits the site at 5:52pm (rounding up), then it won’t run until 6pm. That’s generally fine, but HealthChecks.io doesn’t really understand that. More to the point, I’m guestimating when

    HealthChecks.io has three ways to check time: Simple, Cron, and onCalendar. In general, I use Cron because while it’s cryptic, I understand it. That said, there’s no decent library to convert seconds (which is what WP uses to store the interval timing) which means you end up with a mess of if checks.

    A Mess of Checks

    First, pick a decent ‘default’ (I picked every hour).

    1. If the interval in seconds is not a multiple of 60, use the default.
    2. If the interval is less than 60 seconds, run every minute.
    3. Divide seconds by 60 to get minutes.
    4. If the interval in minutes is not a multiple of 60, use the default.
    5. If the interval is less than an hour (1 to 59 minutes), run every x minutes.
    6. Divide minutes by 60 to get hours.
    7. If the interval in hours is not an even number of days (divide hours by 24), use the default
    8. If the interval is less than a day (1 to 23 hours), run every X hours.
    9. Divide hours by 24 to get days.
    10. If the days interval is not a multiple of 7 , use the default.
    11. If the interval is less than a week (1 to 6 days), run every X days.
    12. Divide days by 7 to get weeks.
    13. If the interval is a week, run every week on ‘today’ at 00:00

    You see where this is going.

    And then there’s the worse part. After you’ve done all this, you have to tweak it.

    Tweaking Timing

    Why do I have to tweak it? Well for example, let’s look at the check for expired transients:

    if ( ! wp_next_scheduled( 'delete_expired_transients' ) && ! wp_installing() ) {
    	wp_schedule_event( time(), 'daily', 'delete_expired_transients' );
    }
    

    This runs every day. Okay, but I don’t know exactly when it’ll run, just that I expect it to run daily. Using my logic above, the cron time would be 0 0 * * * which means … every day at midnight server time.

    But, like I said, I don’t actually know if it’ll run at midnight. In fact, it probably won’t! So I have to setup a grace period. Since I don’t know when in 24 hours something will run, I set it to 2.5 times the interval. If the interval runs every day, then I consider it a fail if it doesn’t run every two days and change.

    I really hate that, but it’s the best workaround I have at the moment.

    Should You Do This?

    Honestly?

    No.

    It’s positively ridiculous to have done in the first place, and I consider it more of a Proof of Concept than anything else. With the way WP handles cron and scheduling, too, it’s just a total pain in the backside to make this work without triggering alerts all the time!

    But at the same time, it does give you a lot more insight into what your site is doing, and when it’s not doing what it should be doing! In fact, this is how I found out that my Redis cache had held on to cron jobs from plugins long since removed!

    There are benefits, but most of the time this is nothing anyone needs.

  • Cute Bears, Uptime Kuma, and Docker

    Cute Bears, Uptime Kuma, and Docker

    I have a confession.

    I use Docker on my laptop all the time to create a stable test environment that I can use and abuse and validate before I push to my staging servers. When it’s just WordPress, I use LocalWP which is hands down one of the best ‘just WP’ desktop tools out there.

    But I don’t really do Docker on my servers.

    Or at least, I didn’t until last week.

    Vibe Coding

    I have a new habit, where I spin up test things while sprawled on my couch watching TV and messing on my iPad. All of this was done on my iPad using:

    • GitHub App
    • Terminus
    • ChatGPT

    Oh.

    Yeah. I used ChatGPT.

    Before you judge me, I validated and tested everything and didn’t blindly trust it, but honestly I did use it for a fast lookup where I didn’t want to figure out the specific search to get to my answer.

    My coworkers joked I’ve gone beyond vibe coding with this.

    Uptime Kuma

    Uptime Kuma is a replacement for UptimeRobot.

    Kuma (クマ/熊) means bear 🐻 in Japanese.

    A little bear is watching your website.🐻🐻🐻

    I mean come on, how could I not?

    Anyway. How do I install this with Docker?

    First of all, I have a dedicated server at DreamHost, which allows me to install Docker.

    Assuming you did that, pull the image down: docker pull louislam/uptime-kuma

    I store my docker stuff in /root/docker/ in subfolders, but you can do it wherever. Some people like to use /opt/uptime-kuma/ for example. Wherever you store it, you’ll need a docker-compose.yml file:

    version: "3"
    services:
      uptime-kuma:
        image: louislam/uptime-kuma:latest
        container_name: uptime-kuma
        restart: unless-stopped
        ports:
          - "3001:3001"
        volumes:
          - ./data:/app/data
        environment:
          - TZ=America/Los_Angeles
    
    

    Keep in mind that data folder? It’ll be created in your uptime-kuma folder. In it are things like the logos I uploaded, which is cool. Anyway, once you’re done, make sure you’re in that uptime-kuma folder and run docker-compose up -d

    Why Docker? Why Now?

    Normally my directions on how you do all this stuff is hella long and complex. There’s a reason I went with Docker when I’ve been avoiding it for (well) years on my servers.

    First of all, it’s isolated. This means it keeps its packages to itself, and I don’t have to worry about the requirements for app A messing up app B. This is hugely important the more apps you have on a server. I have Meilesearch, Uptime Kuma, and more! This gets unwieldy pretty fast.

    The other big reason is … it’s easy. I mean, come on, it’s two commands and a config file!? Compare that to the manual steps which (for one app I manage) can have pages of documentation and screenshots.

    Speaking of easy? Let’s say an upgrade went tits up on your server. Guess what? Rolling back on Docker is super easy. You basically just change the image. All your data is fine.

    I know, right? It’s super weird! But if you remember that data folder? Right so that doesn’t get deleted. It lives in my /root/docker/uptime-kuma/ folder and even if I change the docker image, that folder has my data!

    What’s the Gotcha?

    There is always a Gotcha. In my case, it’s that Nginx can be a bit of a jerk. You have to set up a thing called proxy_pass:

    server {
        server_name status.ipstenu.com;
    
        location / {
            proxy_pass http://localhost:3001;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_redirect off;
        }
    
        listen 443 ssl;
        ssl_certificate /etc/letsencrypt/live/status.ipstenu.com/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/status.ipstenu.com/privkey.pem;
        include /etc/letsencrypt/options-ssl-nginx.conf;
        ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;
    }
    
    server {
        if ($host = status.ipstenu.com) {
            return 301 https://$host$request_uri;
        }
    
        server_name status.ipstenu.com;
        listen 80;
        return 404;
    }
    

    But see here’s where it’s extra weird.

    Now, if you go to status.ipstenu.com you’ll get a dashboard login. Won’t help you, right? But if you go to (for example) status.ipstenu.com/status/lwtv you’ll see the status page for LezWatch.TV.

    And if you go to status.lezwatchtv.com …. You see the same thing.

    The Fifth Doctor, from Doctor Who, looking surprised with a caption of "Whaaaaa!?"

    Get this. Do the same thing you did for the main nginx conf file, but change your location part:

        location / {
            proxy_pass http://127.0.0.1:3001;
            proxy_http_version 1.1;
    
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
        }
    

    That’s it. Take a bow.

    Docker for Everything?

    No.

    I didn’t do docker for my Meilisearch UI install because it’s a simple webpage that doesn’t need it (and yes, that means I did redo my Meilisearch install as Docker).

    I wouldn’t do my WordPress site as Docker … yet … mostly because it’s not an official Docker, but also because WordPress is a basic PHP app. Same as FreshRSS. They don’t need Docker, they need a basic web server, and the risks of compatibility issues are low.

    ChatGPT / AI Coding for Everything?

    No.

    I see where ‘AI’ is going and while I don’t consider it actually artificial intelligence, it is a super complex language learning model than can take your presets and assist you.

    But AI hallucinates. A lot. Like, I asked it to help me set up Meilisearch UI in docker, only to find out that only works for dev and it wanted me to hack the app. I have another project where it’s constantly telling me there’s a whole library that doesn’t exist (and never has), that will solve my problems.

    It got so bad, my boss tried it on his early-release version of an AI tool and it got worse.

    And finally … sometimes it gets super obsessive about the wrong thing. I had a config wrong for Kuma at one point, and ChatGPT kept telling me to check my nginx settings. I had to tell it “Someone will die if you ask me about my nginx settings again” to make it stop.

    What Can I Do with Kuma?

    That little bear is handling 80% of my monitoring now. The remaining 20% are cron jobs that I use HealthChecks.io for (self hosted, of course).

    What are my Kuma-chans?

    • 2 basic “Is this site up?” for GitHub and a service we use.
    • 3 slightly more complex “Is this site up and how’s the SSL cert?” for all three domains I own in this case.
    • 1 basic “Does this keyword exist on this URL?” check for making sure my site isn’t hacked.
    • 2 basic “Does this API key pair exist with this specific data?” for two APIs that do very different things.
    • 1 auth login “Do I still have access to this API?” check for a service.

    I mentioned there are 2 basic API checks, but they do different things. Here’s where it’s fun.

    Ready?

    Screenshot showing I'm checking the LWTV api for last death and confirming the died date.
    Screenshot

    Now that part is pretty basic. Right? Check the API, confirm the date for ‘died’ is what I think, done. And if it’s not? What do I do?

    Well I send a Slack Message:

    Notification setup for Slack where the LWTV Death Toll (this is a joke) tells us when the death is down. it's not really down.
    Screenshot

    This tells us the death is down. Which means ‘someone new has been added to the site as a most recent death.’

    Right now I have to manually go in and change the value to the valid one, but it works. And it’s one way to keep everyone updated.