Half-Elf on Tech

Thoughts From a Professional Lesbian

Author: Ipstenu (Mika Epstein)

  • Too Much Oversight

    Too Much Oversight

    O RLY?People who follow me on twitter know my frustration with my day job.  It’s not that I hate my job, it’s that I hate when the rules get in the way of things.  Over the last three years, we’ve grown from a simple ‘do this please’ directive to a behemoth of monitoring and oversight.

    Here’s an example.  We run standard installs at 3pm once a week. Tickets must have a start time of 3pm and an end time of 5pm. Thank you auditing. If they don’t, they must have secondary approval to give us the okay to go at a ‘non standard time.’

    Now, there is a sane reason for this. We do the install at 3pm, but from 1 to 2pm, we do server maintenance, and from 5 to 9 we do the databases. So really, 3 to 5pm on that one day makes sense, right?  We don’t want to run over or the database guys get mad, and we don’t go early cause the server guys get mad.  We’ve been doing this on the same day, except Thanksgiving or the random ‘on Thursday’ holiday, year in and out for over 30 years. Yes, 30.  Some changes go at 10pm to 2am that night, but the 3pm run for this particular type of change is as normal as anything.

    One day I get a ticket with the time ‘3:15pm to 5pm.’  You’d think I could just say ‘Sure, not a problem.’  It’s within the 3-5 time slot, and fifteen minutes is nothing.  But no.  No, I have to say “I’m sorry, but your ticket requires a start time of 3pm. We are not permitted to make exceptions on this.”

    It burns at my very soul to have to tell people something this idiotic. I mean, it’s fifteen minutes and it would still run within the allotted time! Heck, the process this guy wanted takes 5 minutes total! But no, our tool locks things down to the point that I can neither start the process early nor can I accept a non-standard time without triggering alerts that, at the end of the month, slap me into the “Oversight Review Board” meeting, where I have to explain why I did it.

    DetectiveThe problem is that the oversight machine gets in the way of our ability to be productive.  This mechanism grew from the ‘old days’ when we would submit a request to make a change, and if I didn’t know that the server was being worked on from 1pm to 2pm, I’d just run the ticket whenever.  The timeslots were general guidelines, not set in stone.  Then we grew, and people realized they needed to coordinate a server change with a code push (my job) with a db upgrade and then with some other totally separate install.  And since no one could possibly be expected to memorize every single moving part in the company, we have a new ticket system to manage it for us.

    Old Way: I put in a ticket to make a change with the time/date I’d like it to happen and my boss approves it.  The people making the change pick up the ticket and do the work.

    New Way:  I put in a ticket to make a change with the time/date I’d like it to happen and my boss approves it.  If this change has any red-flags (like it’ll take more than 24 hours, or it affects XYZ), it goes to the Change Review Board, who looks at it and either approves it or asks me to come in and explain what I’m doing.  Furthermore, if I go on certain dates, it goes to another level of review.  If I want to do it in less than a week prep time, it gets extra review and my boss’s boss has to approve it.

    Conceptually, this is meant to have enough eyes looking at a change that someone says “Wait!  Bob, we can’t upgrade the DB severs that day!  Joanne’s major install is that day!”  However, nowhere in here is the system actually checking for us and saying ‘You’re going to be touching the following servers.’  Nowhere does a computer do the mind-numbingly boring work it’s great at and verify that all the interlocking pieces related to my change are also not changing, or if they are, it’s a related change.

    We didn’t make the system work any better, we just became better at covering our asses. Now we know how to write a request with the right buzzwords.  Like every request I make requires me to include what I’m changing, why I’m changing it, how I’ll test it, what documentation is there, who will be the ‘point’ person, what follow up we’ll do, and what unexpected problems might there be and how to we plan to fix them?

    That last one makes me wince.  I often write “We don’t expect any problems, but we’ll follow standard troubleshooting guidelines to fix them.”  My boss tends to have to rewrite that for me, because my capacity for handling stupid questions is usually filled by the time I’ve completed the 10 questions on the form.

    Eye holesI know that the purpose of all this is to make sure that every change we make is one we needed to make, and that it’s done with the right amount of forethought and understanding. What it’s done was make everyone annoyed, and annoyed people don’t do work efficiently. Also it’s asking technical people to write explanations to non-techs, something a number of them aren’t good at and that’s okay! We can’t be expected to be Renaissance Geeks, good at all things.

    Should the technical people be able to say ‘This change will make our ATMs faster’? Of course. And they do. But when they’re asked to detail out every single step, multiple times, in multiple ways, they get annoyed. Instead of asking the question once, they ask the ‘what are you doing?’ question 10 times, in 10 ways, to try and get you to answer what they want to know. And at the end of the day, they still don’t know.

    Of course, the real reason for all this is so that when it goes wrong, the Bobs can point and go “Well, Joanne there screwed up.” and Joanne can point back and say “I said I was rebooting the ATMs at 4am, and you approved it.” and round and round it goes. I made a lot of friends once when I stepped into a M&M(M&M stands for “morbidity and mortality” and is a periodic conference in many medical centers usually held to review cases with poor or avoidable outcomes.) and announced “I can’t see why the system didn’t run as intended, so the logical reason for the outage was that I made a human error and clicked the wrong button.” Of course then they wanted me to code out human error and I decided they were idiots.

    We went from pretty much no oversight past a rubber stamp, and relying on the little guy doing to work to raise any red flags, to massive amounts of oversight where we still rely heavily on the little guy doing the work to raise that red flag. The system locks us in, brokering no room for typos without having to restart the whole chain of events over again, so if you accidentally type in 3:01pm, and the little guy doesn’t notice, you both end up being asked why you did something ‘wrong’ on the metrics report at the end of the month.

    Sometimes in my other posts I say that my perspective on the machinations of things like WordPress and Drupal oversight is different. This is why. I’ve seen the extremes on both ends, and I respect the need for both oversight and attentive management. I think that Open Source tends to handle it better because they can’t afford the big massive teams who have but one job, and that is to know everything. They know they can’t, so they know how to work together. They’re not afraid to email/IM/Skype each other for help, and if everything breaks, they can fix it and laugh about it over beer.

    It’s not that they don’t ‘get it’, it’s that they do get it. Corporate America doesn’t.

  • Said It Once Before But It Bears Repeating

    Said It Once Before But It Bears Repeating

    KeyboardA lot of the time, we complain “Don’t people look in the forums before they ask a question?”  Sometimes we kvetch that these people are ignorant or lazy, and many times they are.  But while a lot of questions are repeated, it’s really not as cut and dried as it may seem.

    We’ve all had conversations where it devolves into “Hey, can you get me that thing?”  Usually this involves a hand gesture or two.  I’m sure everyone’s been in the kitchen with family and pointed right at a cutting board and asked “Pass me that … that thing.”  It’s not that you’re stupid, or ignorant, or lazy, it’s just that right now you forgot the word “cutting board” and as embarrassing as that is in the moment, we all accept that this happens to everyone.

    What does this have to do with questions?  It’s really a language thing, on many levels.  If you ask six different people “What do you call those boxes on the admin dashboard of my CMS?” you’re going to get six different answers. (WordPress had this not too long ago, and it varies from ‘meta boxes’ to ‘widgets’ and everywhere in between.)  Certainly we can attribute this to ‘different brains, different thought patterns.’  But also, if you’ve ever tried to learn a new language, you remember the brute memorization of words than you had to combine with actually using the words, otherwise you forgot them.  It’s like that algebra you did in school.  If you’re not using it, it’ll fall right out of your head.  Unless it’s the lyrics to that song you hated.

    Now that we all accept that we all forget, what’s the deal with not being able to find what we need?  It comes back to naming.  I often complain that looking for an image on Google would be easier if I could draw it.  I’m a pretty visual person, and I like to see what it is that’s wrong.  I ask people “Can you put up a screenshot of the error?” or What page are you on?”  You get a lot of weird error reports in my life, like “When I upload a file I get an error.”

    The problem is not that people don’t know how to explain what’s wrong.  The problem is they don’t know the words to use.

    BinaryOkay, maybe it’s a hair splitting, but when you say “I want my code to do THIS and instead it does THAT.” you will often see yourself using very non-technical terms.  First and foremost, that’s okay.  In fact, I encourage people to use the terms they’re familiar with, that will make it easier to get help.  But you need to know what you’re asking for, and that’s a problem.

    For example.  You’re brand new to WordPress and you want to know how to upload an image.  So you google “Upload image WordPress” and thankfully the very first hit is what you want.  However, the more ‘specialized’ a question gets, the harder it is for the uninitiated to ask it. “I want to change those menu things on the side on my webpage” you think.  Not a weird or uncommon request.  Except ‘menu’ is the wrong word.  You’re really asking “How do I edit my sidebar?” and that’s fairly easy.

    You have to know what you’re asking for in order to get the right help, and you can’t know what it is until you ask someone who knows.  And worse, you can’t know who to ask until you’re familiar with a product and it’s terminology.(Unless you know someone who speaks hand-gesture.)  Basically when you’re learning a new program, you’re learning a new language.  Worse, you’re learning a whole new culture.  It’s like being back in college, where everything is hugely different from High School.

    That’s why I try very hard not to get frustrated when I see the same question a million times.  It’s rarely worded the exact same way.  However, when I see people ask “How do I add images to WordPress?”  I wonder if they’ve heard of searching.

    The steps are easy:

    1. Look it up on Google/Bing/whatever
    2. Search the forums
    3. Ask

    When you hit step three, you have to ask it in the clearest language you can.

    On my site, I have two ‘sections.’  There’s the main part where all my posts are, and there’s this list on the side that has calender, login links, and other stuff.  Here’s what it looks like: .

    Now you’ve asked a very clear question.  Let’s compare it to the ones I saw recently:

    My site has this stuff I don’t want. How do I remove it?

    Which one would you answer?

    It’s okay not to know what you’re asking about in detail.  No one is born knowing, and you have to learn somehow.  Accepting the fact that you’re new, or ignorant at something, is hard.  We’re often told it’s not okay to not know things, that being ignorant is a sign of weakness. That’s just not true. We’re all ignorant and we’re all new about something. Everyone starts out ignorant. And we all start out not even knowing how to ask a question.(If you’ve ever been to a Passover dinner, the point of the Four Questions is to remind us that we all start out as the simple child who doesn’t know how to ask a question, and thus we tell a story every year to teach them, and ourselves.)  The problem is that the answers you get depend entirely on how you ask the question!  If you ask it wrong, you get the wrong answers, which frustrates everyone.

    Should you search first and then ask? Of course. But you, as the helper, need to remember that the language barrier is going to cause a lot of the repetition. You’re going to get the same question, something basic and simple, asked a hundred times because people don’t know what they’re really asking. And yes, if there’s a good tutorial, toss them a link. But remember these people are new and scared and tossing them a link doesn’t help because they still may not speak the lingua fraca of the help docs!

    Language

    Step back.

    Remember what it was like to be new, and you’ll understand why the same questions are asked a hundred times over. Point them to the documentation. If the docs are lacking, fix that and make them better for the new guy. And, as hard as it is, especially right now in this holiday season, don’t get mad. Take a deep breath, watch Tron or play a round of World of Warcraft (or Mario Kart Wii) and remember that first time you did something, and how hard it was to learn all those terms and understand what they meant.

    And you as the asker?  You need to remember that sometimes, when the helpers ask you something that seems silly or weird, or overly simplistic, there’s a reason.  We like to take apart a complicated question to its simplest iteration in order to understand what it is you want, and to be able to teach you “Okay, when you say ‘non blog index’ what you mean is a ‘static front page’ which is ….”  We want you to learn our lingo so we know that we’re all speaking the same language.  Otherwise we’re just doing a modern rendition of “Who’s On First.”

  • WordPress Sidebars as Menus: Part 2

    WordPress Sidebars as Menus: Part 2

    Happy Thanksgiving.  Here are some more ideas, partly based on the comments left in post #1.  At the bottom is a gallery of all the various mockups, and feel free to download, tweak, etc.

    More Compact

    Instead of a big Sidebar Locations box in upper left, what if you made location an element in the Sidebars themselves (Primary, Test)?

    This has the location selection in the Sidebar Area itself.  I’m not sure if I like the multiple saves, but if you have a long Sidebar Area, it seems sensible I made the space a big bigger with the idea that plugins could hook in and add things.  More on that in a minute.  You’ll also notice that there’s a scrollbar for the ‘Available Widgets’.  Yeah, we lose drag/drop with this scenario, and while I agree D&D is very cool, it’s starting to get unmanageable when you want to drag a widget over to the area halfway down the screen.  My grandmother said it was impossible for her to scroll in two directions (over and down) while holding down the mouse button.  Mind you, she’s a 90-year-old with glaucoma.

    Selecting a Location

    Here’s what the dropdown looks like.  Obviously we’re on Twenty Eleven here.  The ‘blank’ is for ‘none’ which, on reflection, may need to become ‘(none)’ instead.  It’s obvious to me that blank == none, but I’m not sure how new users would feel about that.  Yes, my rounded corners suck.

    Hover Over

    Again, my colors  and icons suck here, but this is a large pointer finger hovering over Custom Menu being told “Use this to….”  My (minor) concern with this is that Akismet, for example, has the description of … Akismet.  Singularly useless.  You’d think it’d be better.  But they’re not the only ones who slacked off on descriptions, so some of these will suck.  Still, color it any which way and a hover-up will provide information.

    The major concern I have, again, is accessibility. I’m hoping that this has already been hashed out before and we don’t have to invent something all new to allow screen readers to parse what things are for.  That would be a deal breaker to me.

    New Sidebar Screen

    Here is the ‘new’ screen, complete with directions and a button.

    Jane's Suggestion - Green

    By the way, since I suck at gradients, I opted not to make ‘Create Sidebar’ in the button, but that’s a nice idea too.  Of course, with Jane’s recent post about the square button, that  works too.  I spun up a inverse of that, since it mimics the blue background and looks ‘obvious’ to me.

    With Description

    I LOVE the idea of a visual guide to where the sidebars are. Makes me think of Stephanie Leary’s layout fiddling with IDs:http://sillybean.net/downloads/widget-admin-ui-altered-with-ID.png (Trac ticket #18334:http://core.trac.wordpress.org/ticket/18334 – some other cool ideas there, too.)

    I love Stephanie’s idea too, but. I really didn’t like the ‘uneven’ feel of her screen (not her fault, it’s just CSS layouts drive people to drink).  Her’s works because you see where the widgets are going to go.  I would want to have it be a wireframe.  This is one idea for where to show the description, though I’m not really sold on it.

    Sidebar Logic

    know that Jane mentioned per-page widgets as a priority (maybe in IRC?), but we’ll have to wait until after the core team meet up to see what they decide on as goals.

    On the other hand, I really like my idea for Sidebar Logic.  If you’ve used Widget Logic, you get the idea.  Put in the PHP to say ‘This Sidebar shows up on the Main Sidebar area IF these parameters are met.’  It’s not as ‘per page’ as Jane probably had in mind, though, and I’d like to see it avoid the need for PHP, but on the other hand, I’d love to see something plugged in there.

    That’s all I have for you today, but here’s the Gallery, as promised!

  • WordPress Sidebars as Menus: Part 1

    Okay, fine, not all widgets are used in sidebars. I’m going to use sidebar here to make my life simpler and trust you know what I mean. After reading Trac 17979 and Jane’s post on Wherefore Art Thou, Widgets? I had some thoughts.

    Right now, when you switch themes, if the sidebar doesn’t match a new sidebar area, the widgets get dumped into ‘Inactive’, which makes if difficult if you want to switch themes for testing. Kbitzing on this on Twitter, like we do a few of us started kicking around ideas, most of which the UI team folks in dev-chat had already gone through.

    My first draft was pretty straight forward:

    But then Jane pointed out:

    https://twitter.com/#!/janeforshort/status/138400574516375554
    https://twitter.com/#!/janeforshort/status/138400669953564672

    Which is true. I have over 20 available widgets here, and that list would get pretty damn long. While a list of checkboxes strikes me as less pretty, you would be easier able to manage this:

    In either case, you’d want the widgets ‘box’ where they’re listed to be scrollable. I’m fairly sure you could make an auto-resizing box that grew and shrank by height depending on how tall your window is and then allow for scrolling the rest of the way.

    At this point, we’ve reached the end of my practical knowledge. I know that menus are stored in the wp_posts table as a custom post-type called nav_menu_item. And then the actual data is over in the wp_postmeta table, where the post_id is the same as the ID from the posts table, and the _menu_item_* settings are where the magic happens.

    Post Meta Menu Example

    On the other hand, widgets are stored in the table wp_options and not in the same way. They’re in one master field sidebars_widgets which stores all your information for widgets, which ‘sidebar’ they’re in, active or not. Each individual widget stores itself in widget_NAME.

    Widgets in the DB

    Clearly it would be a bad idea to attempt to save the widget data in post-types, so we’d have to have some way to reach back and get the data. But if it were possible to toss the sidebars_widgets data over into the posts table, then they’d be brought over when you ran an export/import of your site much like menus are, which would make moving people off of MultiSite and over to their own site a heck of a lot easier, wouldn’t it?

    This is as far as my thinking takes me, but it’s something I’d love to play with.

  • How Likely Is It That My Upgrade Will Fail?

    How Likely Is It That My Upgrade Will Fail?

    My father, Woody, is a risk analyst. So I asked him, knowing my math skills, where should I start learning about how to analyse and assess risk. With the personal commentary removed, here’s his answer.

    Math is not very important, at least not at the beginning. Risk assessment is really just thinking hard about answers to the 3 fundemental questions: what can go wrong; how likely is it; what are the consequences?

    Look at what you do at work. How can good answers to the three questions mitigate the (bad) consequences of poor decisions?

    Do a pilot study with an up-coming decision.

    Remember that what can go wrong? means an analysis from a choice or intiating event (like a 3-day power black-out in Chicago) of the sequence of events and failures of systems to control the events, bad human decisions, etc. Each sequence ends up in a bad situation or an ok situation. How likely is it? is just the likelihood of that sequence occuring, usually measured by a probability for each event in the sequence, either through data or expert judgement. What are the consequences? means that for every bad ending of a sequence, what are the consequences of that bad state.

    Make a decision-tree or event tree to enumerate the sequences. Each branching point (or top event) can have a fault tree to represent how that branch point fails or succeeds, or just expert judgement.

    Represent the likelihood of failure as a number between 0 and 1 (then success will equal 1-failure).

    Choose an end state for each sequence. Multiply the numbers for each branch point to get the likelihood of the sequence.

    Add up all the sequence likelihoods for the same end-state.

    That’s all there is to it.

    When you put it that way, it does look pretty simple.

    So I went through a proof of contcept process.  This is my first time making a fault tree, and I didn’t bounce it off my father.

    Fault Tree of a WP Upgrade

    As you can see, this is pretty basic. What can go wrong? A lot actually, and I wasn’t really doing more than picking the common problems. But this is a fault tree, not a decision tree. Are they different? They are! A fault tree is basically what you use to suss out why things go wrong. A decision tree though, we make a list of decisions and spin out what the liklihood of a failure is. So my decision here is “How should I upgrade WP? Stable or Pre-Release?

    WP Upgrade - Decision Tree

    Here you’ll see this is a similar enough, but wait! I have funny numbers! That’s my guesstimate at how likely these are to cause problems. See if you don’t have high tech skills, using SVN to upgrade is higher risk. In this world, you want a lower number. Like if you look at the stable release, you’d see that it adds up to a .4 failure, or a 1% that it’ll fail because of the upgrade tool or the user’s tech skills, but a higher 2% for ‘breaks’ (by which I mean you have a crappy plugin or theme).

    Now I left off things like for SVN/Nightly/Beta/RC you get the cool toys early, mostly for space and since this is a poof of concept. It’s clear that SVN is something only experienced people should play with, but it’s very possible I’ve scored Beta/RC too high. They’re sort of a break-even point, though. While Stable will always be recommended, I did a quick revamp of Nightly and Beta/RC. Nightly’s are more risky because you run a risk of getting an incomplete build (that is, some of my bored maniac friends may be checking in code, and not be 100% done when you run your update – a common weird issue with SVN and why I always svn up before I consider reporting a bug). But a Beta/RC is a ‘very nearly done’ cake, just missing the icing.

    WP Upgrade - Decision Tree Take 2

    Version two is, you can see, very similar. Personally I consider this a ‘start’ to understanding the risks inherent in a WordPress upgrade. If you held a gun on me and demanded I explain where I got the numbers, I would call them educated guesses, based on the forums, the mailing lists and my personal experience. Dad would say ‘Expert Judgement.’

    My next steps are to read up more on the process of using decision trees, directly in relation with software. While I certainly will also be looking into how a tornado in downtown Chicago would impact my office (can I get to work? No? Okay, so VPN. Can it take 5,000 people at once? Based on Snowmageddon last year, no. etc etc and so on), understanding the logic trees behind the forms is always my first step.

    To my WordPress friends, please let me know if I scored things too high or low in this one! To the rest of you, if you use these sorts of things in your jobs and, if so, how. I’d love to see some real-world applications outside the financial world!

  • WordPress, DSO and Permissions

    WordPress, DSO and Permissions

    I run my server with PHP DSO.(For the differences between DSO and SuPHP, read DSO (mod_php) vs. CGI vs. suPHP vs. FastCGI) It lets me run APC, and I’ve always liked it. It does have some weird problems, mind you, like a tendency to upload files as nobody:nobody, and more importantly it means that you have to set your wp-content/uploads folder permissions to 777. Thankfully there’s a fix!

    If you’re not good with command line, scared by shell, and terrified of chmod, you’ll need to find your friendly neighborhood sysadmin to help you out. It’s okay to not feel up to doing this, and it should go without saying that you should make a backup first!

    To step back, someone’s going to ask “Why is 777 bad?” Unix permissions are complicated. Every file in UNIX has an owner user and an owner group, and most of the time they’re the same. Mine are ipstenu:ipstenu (which means owner ipstenu, group ipstenu). Now another account on this server, conrel, has conrel:conrel. The groups ipstenu and conrel are both in the same webmaster group, which gives them special permissions. It’s confusing to a lot of people that most webhosts use the same name for the user and the group, but it’s just what we do.

    Now for every file, there are three types of ‘ownership’:

    1. User ownership – i.e. the user ipstenu
    2. Group ownership – i.e. the group ipstenu
    3. No ownership – i.e. you who are reading my site

    There are also three types of permission levels”

    • read (r)
    • modify/edit/write (w)
    • execute/run (x)

    This all works out so when you go in via unix shell and look at your files you see soemthing like this:

    -rw-r--r-- 1 ipstenu ipstenu 203789 Oct 5 19:30 stevejobs.png

    This means the owner (ipstenu) has rw permissions (which are read-write). The group (ipstenu) has r (read-only), and the world (i.e. everyone else) also has r (read-only). This is an image, no one needs to execute it (which would be an x).(The “1” before ipstenu is for the number of files. “203789” is the size of the file. “Oct 5 19:30” is the day/time I uploaded the file, and “stevejobs.png” is the name of the file.) These rwx letters correspond to numbers. r = 4, w = 2 and x = 1. So when you see ‘rwx’ that equals 7.(There are also options o (other), u (user), g (group) and a (all)… and s … but I’ll spare you that right now. Suffice to say, you can use what you’re comfortable with. I use the numbers most of the time.)

    So why is 777 dangerous? 777 means ‘everyone has full access to this file.’ Yeah, that sounds dangerous! I don’t want that! The only person who should have full access is you! But DSO doesn’t like to upload files without 777 permissions. In part, this is WordPress’s fault, but really it’s an unholy combination of things. Alex King explains why it happens, and as of WordPress 2.8, you can fix this yourself.

    Just override the default file permissions. It’s genius! I tossed this into my wp-config.php file and I was good to go!

    define('FS_CHMOD_DIR', (0755 & ~ umask()));
    define('FS_CHMOD_FILE', (0644 & ~ umask()));
    

    No, the 0 in front is not a typo. 0755 is an octal value. Octal values must be prefixed with a 0 and are not delineated with single quotes (‘). It’s just how it works.

    There is a catch, though. My uploads folder had been set to 777, which meant /wp-content/uploads/2011/10 (this month’s folder) was also 777, which totally invalidated my test. That’s easy enough to go back and fix permissions on your folders. I did it this way because I have some caching plugins that I do not want to screw around with:

    find /home/foobar/public_html/wp-content/uploads -type d -perm 777 -print -exec chmod 755 {} \;
    
    find /home/foobar/public_html/wp-content/themes -type d -perm 777 -print -exec chmod 755 {} \;
    
    find /home/foobar/public_html/wp-content/plugins -type d -perm 777 -print -exec chmod 755 {} \;
    

    That code says “Find all folders (-type d) and if they have permissions of 777, change them to 755.” There are more variations on that.(I got the code from NixCraft – Linux / UNIX: Change File Permissions Recursively ( conditional )) If you want to change files, it’s -type f and you’d want something like this:

    find /home/foobar/public_html/wp-content/uploads -type f -perm 777 -print -exec chmod 644 {} \;
    

    That will turn all your images back into permissions 644, presuming they were 777 to begin with. Mine were 755.

    Permissions GrantedThe last step I had was chowning the folder for uploads and 2011 to nobody:nobody. That was so on month end, I would be able to create folders (like uploads/2011/11 today) without any issues. The other folders, as they already existed, didn’t need the permissions changed. Honestly, I’m not sure if I needed to set the uploads folder to that. I didn’t set blogs.dir for my MultiSite install, and just did the files folder within, since it had created other folders correctly. It’s a hassle, unraveling years of ‘Did it wrong!’ and when you add in that we’re using different tool sets to upload files versus upgrade and all that … well. It works now.

    I also kept the upgrade folder with permissions 777, since that just did not want to work any other way. It flat out refused to upgrade any plugins. I’ve yet to try upgrading WordPress itself with this setup, but I suppose I’ll find out soon.

    And that’s it! It’s not 100% painless, and it’s much easier if you start out ‘doin’ it right’, but even after you’ve been doing it wrong for over 5 years, you can fix it.