Любовное письмо к Clojure, часть 1

(Между прочим, люди сообщали о похожих чувствах ужаса после прочтения книги Брайана Гетца « Параллелизм на практике на Java», осознавая все вещи, которые могут пойти не так в параллельном программировании, в написанном ими коде, часто приводя к тому, что он охватывает такие языки, как Clojure. Подробнее о Брайане. Гетц позже.)

В течение следующего года меня осенило, как запрет на переменную мутацию делает программы проще и понятнее даже для однопоточных программ. Это заставило меня осознать, что «передача переменных по ссылке», что, как мне показалось, экономит время, на самом деле было одной из причин, которые вызывали у меня огромные проблемы, потому что это приводило к изменению переменных в удаленных частях кода.

Вот короткое видео, которое я сделал, в котором описан мой ага момент, когда потенциально опасные переменные по ссылке относятся к ничего не подозревающим.

Страшилки! После программирования в Clojure это казалось настолько опасным (и вульгарным!), Что вы могли вызывать функцию, и это могло изменить объект, который вы ей передали! Как вы можете рассуждать о программе, которая допускает такие неконтролируемые мутации?

Так же, как Clojure показал мне, как более простой синтаксис LISP освобождает ваш мозг, чтобы больше думать о проблеме, которую вы хотите решить. Мир без мутаций заставил меня понять, насколько трудно отслеживать, как происходят мутации в ваших программах.

Но не принимайте это от меня — возьмите это от Джона Кармака , который повлиял на многих из нас, как одного из основателей id Software (DOOM, Quake и т. Д.), В настоящее время технического директора Oculus VR. В 2013 году он написал удивительную статью в журнале Gamasutra о возможности использования концепций функционального программирования на C ++.

Мое прагматическое резюме: большая часть недостатков в разработке программного обеспечения связана с тем, что программисты не до конца понимают все возможные состояния, в которых может выполняться их код. В многопоточной среде отсутствие понимания и возникающие в результате проблемы значительно усиливаются почти до точка паники, если вы обращаете внимание. Программирование в функциональном стиле делает состояние, представленное вашему коду, явным, что значительно упрощает рассуждение, а в полностью чистой системе делает невозможным состояние гонки потоков.

No matter what language you work in, programming in a functional style provides benefits. You should do it whenever it is convenient, and you should think hard about the decision when it isn’t convenient. You can learn about lambdas, monads, currying, composing lazily evaluated functions on infinite sets, and all the other aspects of explicitly functionally oriented languages later if you choose.

C++ doesn’t encourage functional programming, but it doesn’t prevent you from doing it…

Pure functions are trivial to test; the tests look like something right out of a textbook, where you build some inputs and look at the output. Whenever I come across a finicky looking bit of code now, I split it out into a separate pure function and write tests for it. Frighteningly, I often find something wrong in these cases, which means I’m probably not casting a wide enough net.

I remember reading this article in 2013, but it wasn’t until I re-read it in 2016 that I realized how universal this problem is. It’s not just for games written in C++ that need to run at 60 frames per second; it’s for any programmer.

In his 2013 QuakeCon keynote, he describes his experiments rewriting Castle Wolfenstein 3D using functional programming techniques in Haskell to explore to what extent one could use a programming language that disallowed mutation, which was absolutely fascinating.

(In this talk, Carmack also talks about his explorations of LISP, doing the exercises from the famous MIT SICP book, why he’s growing ever more convinced on the value of strong, statically-typed languages for large codebases that need to be maintained for a long time, and so many other fascinating observations from one of the programmer legends… I will be writing more on this in Part 2.)

Composability: A Mistake I’ve Been Making Since Grad School

After gaining more proficiency in Clojure, many other things become apparent. One thing that I learned is the benefits of pushing side-effects (i.e., any impure functions, such as anything that involves input or output) to the edges of the program and making sure that everything else in between is a pure function.

To recap, pure functions are those that the output is strictly a function of the inputs. One of the benefits of pure functions is that the function can be tested completely in isolation, as John Carmack noted above.

When I started making a point of doing this, the benefits started becoming quite apparent. In fact, I started to get the suspicion that not practicing this was one of the main reasons the code I wrote would become increasingly untestable and difficult to change-which led to the feeling that my code was collapsing under its own weight.

This is a problem that I’ve been having for a very long time… In fact, I still have a very vivid memory of when this problem killed a project I was working on. It was during my high speed compilers class in graduate school in 1994, where we built a Modula-2 compiler in C++ that generated SPARC assembly code, which was then compiled into a Solaris executable.

You first wrote a lexical analyzer to tokenize the input (using lex ), then the compiler that would turn the tokens into an abstract syntax tree (using yacc), turn it into some intermediate format, and then emit assembly code. And then use as to compile.

It started off great, especially since I had written lexers and parsers before. But at each later stage, I remember the feeling of tucking the previous compilation phase into the next phase, thinking, «Okay, here goes nothing… I’m putting my code somewhere where I can no longer directly access it, and it feels like throwing it into a deep, dark well… I sure hope it works.»

In other words, I had written the compiler phases in a way where each phase was no longer independently runnable, let alone testable. I was good enough where it was not really a problem until the very last phase, where we were executing our SPARC executable program.

I remember my compiler working for all the test cases, until we had to implement recursion. Then, it started blowing up after a certain number of recursive calls. After several all-nighters, I finally figured out that I was incorrectly computing the memory locations of variables on the stack, probably incrementing the stack pointer the wrong way. After a certain number of recursive function calls, the generated program would just segfault and die.

Even though I figured out conceptually what was going wrong, I ran out of time, and couldn’t fix the mess I had created before submitting my fatally flawed compiler. I probably got one of the lowest grades in the class.

What I had learned was that burying my code, so that it could only be executed inside other functions, violated the principle of composition — the ability to run and test a program, independent and isolated from the other modules. (My thanks to Dr. Stephen Magill for taking the time to explain this to me, who also helped me write my first Haskell program!)

Here’s some pseudo-code of the good way vs. bad way…

 ; ; good way: steps are composed together, which each compiler phase ; indepdendently executable and testable ; (-> (tokenize-source-files!) (generate-abstract-syntax-tree) (generate-intermediate-representation) (generate-assembly-instructions) (write-assembly-output-files!)) ; ; bad way ; all the intermediate steps buried inside other functions, no longer reachable or inspectable ; tokenize-source-files-and-generate-ir-and-generate-assembly();

In Clojure, I fell in love with the convention of marking impure functions with a !, to make it obvious when side-effects could happen. This includes any function that reads from disk or a database, let alone writes to one. After all, even external inputs make the output impure. These effectful functions should be pushed to the beginning or the end of your program, which makes everything in the middle independently runnable and testable.

It’s embarrassing that it’s taken thirty years after I started coding professionally to finally discover this mistake, which has plagued almost every large program that I’ve worked on.

And of course, many people learned this twenty years ago, either by intuition, experience, or by reading great books, such as Refactoring: Improving the Design of Existing Code by Martin Fowler. But knuckle-headed me only learned after being forced to confront this in Clojure, which seems to bring these problems to the front and center.

(What’s also interesting to me is that by pushing side-effects to the edges, almost everything can be rewritten as pure functions and be tested with unit tests. Previously, most of my tests often tested my code and the I/O, API calls, etc… In the ideal, you just want to test the code you wrote. You don’t need to test whether the file system works in every test, or whether Trello API works, etc…. By pushing I/O to the edges, the need for brittle mocks and stubs almost disappears entirely.)

The Epiphany I Had Reading «React for jQuery Programmers To Get By»

Something else happened during my first rewrite of TweetScriber that primed me for another Clojure aha moment. Knowing almost nothing about writing a JavaScript app, I was initially doing some research and stumbled upon this amazing article, An Introduction to React in 2019 (For People Who Know Just Enough jQuery To Get By) which was originally written by Shu Uesugi (@chibicode) in 2015 and revised by Julien Benchetrit (@julienbenc).

I had virtually no experience with working in the browser DOM, had never used jQuery (but had heard about how revolutionary it was years back), and I’ve certainly never used one of these more modern frameworks like React.

The article seemed especially timely because, at first glance, the exercises seemed to cover 50% of the TweetScriber functionality I needed! But reading through the exercises, it seemed to discuss something very, very important, with some profound lessons in how a program’s state leads to incredible complexity — more things that have caused my programs to eventually collapse in on itself.

• Step 1. Write a Tweetbox, which lets you type into an textarea window, with a Tweet button. Cool. I barely knew enough HTML to write this, so copying this code would be awesome. Easy enough.
• Step 2. Change the Tweetbox so that if the contents exceed 140 characters (the original article was written in 2015), then disable the Tweet button. Great. We now have some state associated with the program, but it isn’t too bad, because it’s just keeping track of the number of characters in the textarea, adjusting whether the button is disabled or not.
• Step 7. Change the Tweetbox so that it displays characters remaining, and a display message that indicates by how many characters you’ve exceeded the limit. Okay, this is getting a little more complicated because, now, these different components need to be updated differently based on the state…
• Step 9. Add an Add Photo button, which displays «photo added» if a photo has already been added, and also subtracts 23 characters from the number of characters remaining (because links to photos consume characters). Okay, the point of these exercises was becoming very evident because, now, state management is becoming a real issue, and using the jQuery style of callbacks was becoming a real mess. Even keeping track of which components needed to know of each other, and what their responsibilities are to each other, seemed impossible to remember. And this is for a small toy application!!! In a real application, the complexity is orders of magnitude worse!

In other words, in the jQuery implementation, the flow of information from one component to another becomes entangled. The diagram below shows it magnificently.

In contrast, consider the same functionality implemented with something like React, in a more functional style, where state is mutated in one place, and all the UI elements are rendered from the state. State is changed in a very uniform way, and the UI is rendered as pure functions, never mutating state.

The diagram below show how much simpler the data flow is…

I found this to be a shockingly convincing argument about how code should be written. And I was pretty happy with my first version that I wrote in TypeScript and React.

(In my ClojureScript rewrite using re-frame, it addressed one of the problems I found in using React, which was state is scattered across all the components. Re-frame and Redux take a different approach, which is to put all mutable state into one place, which I’ve found works magnificently. This has been so successful for me that I will never write a client web app in any other type of framework.)

I think this aha moment prepared me for the discovery of Rich Hickey and Clojure…

Rich Hickey on Simplicity: Where The First Ideal Of Locality and Simplicity Comes From

Rich Hickey is the inventor of Clojure, and his talks and Clojure have affected how I think about software. In fact, this entire blog post attempts to frame some of these aha moments.

There was one particular Rich Hickey talk that hit me like a ton of bricks, which was his famous «Simple Made Easy» talk that he gave at Strange Loop 2011. (If the jQuery/React epiphany was an aha moment, this talk was a chorus of aha moments.)

The talk is on the InfoQ site here (which shows the slides associated with the soundtrack), and you can find a wonderful transcript of the talk here. You can also find a version of the talk he did for the Ruby and Ruby on Rails community on YouTube here.

This talk is so impactful to me that I’m almost tempted to say, «Go ahead and take an hour to listen to the talk. I’ll wait here.»

Among many other things, he talks about how limited the human brain is in its ability to reason about things, to keep track of things, to grapple with complexity (like the jQuery example above). He stated that the difference in cognitive capability between your average programmer and your above-average programmer is not the vaunted 10x difference. He said (emphasis mine):

Then we have this other part though, which is the mental capability part. And that’s the part that’s always hard to talk about, the mental capability part because, the fact is, we can learn more things. We actually can’t get much smarter. We’re not going to move; we’re not going to move our brain closer to the complexity. We have to make things near by simplifying them.

But the truth here is not that they’re these super, bright people who can do these amazing things and everybody else is stuck because the juggling analogy is pretty close. Right? The average juggler can do three balls. The most amazing juggler in the world can do, like, 9 balls or 12 or something like that. They can’t do 20 or 100. We’re all very limited. Compared to the complexity we can create, we’re all statistically at the same point in our ability to understand it, which is not very good. So we’re going to have to bring things towards us.

He talks about the need for simplicity in the software we write and the stuff that we write our software in. The conditions for simplicity include having components that are completely decoupled from each other with no knowledge of each other. Just like in the jQuery example, things quickly become a complete mess when all the different controls have to know about each other. It’s difficult to get it to work correctly, to reason about what happens when state changes, and it’s difficult to write additional functionality to it.

Hickey goes through the some of the core concepts in programming languages, and describes the extremes of which represent the ideal of simplicity and which ones represent complexity that eventually leads to the inability to understand and safely change our code, as well as misery and catastrophe.

As I mentioned in the beginning of this article, The Unicorn Project (download the excerpts) is about the invisible structures that enable developers to be productive. So much of this comes from the concepts that Rich Hickey has espoused over the years, and which have become infused into Clojure and the Clojure community.

Here’s an excerpt from The Unicorn Project where Erik summarizes Rich Hickey’s concepts around simplicity and “complectedness.”

Erik answer, “‘Complect’ is an archaic word, resurrected by Sensei Rich Hickey. It is a verb that means to turn something simple into something complex.

“In tightly coupled and complected systems, it’s nearly impossible to change anything, because you can’t just change one area of the code, you must change one hundred, or even a thousand, areas of the code. And even the smallest changes can cause wildly unpredictable effects in distant parts of the system, maybe in something you’ve never even heard of.

“Sensei Hickey would say, ‘think of four strands of yarn that hang independently—that’s a simple system. Now take those same four strands of yarn and braid them together. Now you’ve complected them.’ Both configurations of yarn could fulfill the same engineering goal, but one is dramatically easier to change than the other. In the simple system, you can change one string independently without having to touch the others. Which is very good.”

Erik laughs, “However, in the complected system, when you want to make a change to one strand of yarn, you are forced to change the other three strands, too. In fact, for many things you may want to do, you simply cannot, because everything is so knotted together!

“And when that happens,” he continues, “you’ve trapped yourself in a system of work where you can no longer solve real business problems easily anymore—instead, you’re forced to merely solve puzzles all day, trying to figure out how to make your small change, obstructed by your complected system every step of the way. You must schedule meetings with other teams, try to convince them to change something for you, escalate it to their managers, maybe all the way up the chain.

“Everything you do becomes increasingly distant from the real business problem you’re trying to solve,” he says. “And that, Dwayne, is what everyone discovered when they switched out the routers in those manufacturing plants. Before, you had three independent strands, with team able to work independently but at the cost of having to maintain three networking switches.

“When you put them all on one switch, you complected their value streams, all now having dependencies on each other that didn’t exist before! They must constantly communicate, coordinate, schedule, marshal, sequence, and deconflict their work. They now have an extremely high cost of coordination, which lengthened lead times, decreased quality, and in your story, led to a week-long catastrophe that significantly impaired the business, going all the way up to Steve!” Erik says with glee.

You can find many collections of «Rich Hickey’s Greatest Hits» on the Internet. Here are some:

• https://github.com/tallesl/Rich-Hickey-fanclub.
• https://www.youtube.com/playlist?list=PLXsqD83He-e5oUh_DFrHbO3MoNj3tG8Vh.

In The Unicorn Project, the joke about «how many people do you need to take out to lunch in order to get a feature done» was taken from his talk that he did at 2015 Java One conference: https://www.youtube.com/watch?v=VSdnJDO-xdg.

In fact, in some ways, coupling as one of the key themes of the book came from watching this talk, where he compared modern REST interfaces with the bad old days of CORBA and Sun RPC, where changing anything needed everybody to cooperate.)

What I find so interesting is that functional programming also eliminates iteration and looping, which is so prone to off-by-one errors. In The Unicorn Project, when Maxine pairs with some middle schoolers in Python, she spots their «off by one» error when they’re iterating through an array. This was inspired by Cornelia Davis, describing when she worked with her college-aged son on a genomic sequencing program in Python, showing the contrast in imperative vs. functional programming style: https://youtu.be/R1RDhUf1Go4?t=492.

By the way, here’s a whole brilliant talk about showing how to convert an imperative program into a more functional style in JavaScript: «Solving Problems The Clojure Way,» by Rafal Dittwald.

Solving Business Problems, Not Solving Puzzles: Why I Detest Infrastructure These Days…

So, I feel like I need to explain another consequence of now recognizing and wanting to avoid complexity. I mentioned that I now self-identify as a developer…. What I didn’t tell you was that I now self-identify as one of those very fussy, parochial developers that Ops people hate.

In my ideal, I just want to work on solving the problem I set out to solve, working within my pure functional application bubble. And I don’t want to deal with anything outside of that bubble.

Here’s all the things I used to love doing, but now I detest doing…

• dealing with anything outside of my application.
• connecting to anything to anything (including databases).
• SQL databases and figuring out why my queries are so slow.
• updating dependencies (because, so hard…).
• secrets management (ditto).
• Bash.
• YAML.
• patching (so inconvenient…).
• building kubernetes deployment files (mostly by Googling).
• trying to understand why my cloud costs are so high.

This is not to say that these things aren’t important. On the contrary, they’re absolutely critical. Which is why I think the brightest days of Infrastructure and Operations and Infosec are still ahead of us, not behind us. I believe the job of infrastructure and security people is to create the platforms that developers can use to take care of all these non-functional requirements, without having to Google and Stack Overflow all day.

Lastly, the REPL…the Ultimate In Fast Feedback Loops!

Okay, it’s about time to wrap up this first article, which is pushing 7K words, and there’s so many other things I want to write about because I’m so excited by them, but it’s time to post something…otherwise, I’ll never post anything.

But there’s one thing that I need to write about, which is something that seems pretty unique to Clojure, or LISPs, which is the amazingly interactive nature of development. One reason I think I’ll be using Clojure for years (or decades) to come is because of the REPL.

REPL stands for Read-Eval-Print-Loop, which many languages have. But I don’t know of any other modern language that can be integrated into an editor, where you can modify the running program, inspect variables, run functions…from inside the program!

In the spirit of Focus, Flow, and Joy, I think the REPL experience is absolutely amazing and sublime. It feels like you’re building your program from the inside, writing and testing expressions, saving I/O to variables and transforming the output using pure functions. I’ve done this with Clojure programs running on the JVM, or ClojureScript programs running in a browser… I’ve done it to analyze and visualize large data sets, save to and query databases…

Going back to programming where you have to save and reload your program, and then maybe even navigate to get back to a certain program state, just seems intolerable now.

This is definitely one of those things that you have to see a master at work to fully appreciate. Here’s Bruce Hauman talking in 2015 about the now-famous figwheel, a ClojureScript build tool that enables live code reloading (and so much more). Watch as he shows something he wrote on the plane ride out to the conference, as he changes the solar system simulator and the JavaScript app gets hot-reloaded in real-time.

(Yes, you can do this in JavaScript, but trust me, due to how dicey state mutation is, it’s very flakey. On the other hand, the way Clojure enforces mutation to be done in a very specific way makes it almost perfect for this style of coding.)

Here’s a 2019 video of Sean Corfield (famous for his work on the mostly widely used Clojure JDBC libraries) showing how he uses the REPL to debug a problem reported in the core.memoize function, using the Atom editor and the Chlorine Clojure plug-in.

He walks through the steps required to reproduce the problem, form by form, and the output of each form as it’s evaluated is displayed in the editor. In the failing case, we watch as he inspects the internal state of the function, hypothesizes what is going wrong, confirms it, and fixes it. (We then watch as he creates an additional test to the Clojure test suite and runs in across 10 years of Clojure versions.)

I think Sean makes debugging almost look fun, and it shows just how powerful working within a REPL can be.

(I use the Cursive plug-in for IntelliJ by Colin Fleming and team, which I find to be absolutely terrific. I use Visual Studio Code for almost all non-Clojure files, so I have high hopes for the Calva plug-in… But I love the Cursive features so much I can’t imagine switching anytime in the near future.)

Here’s a great video of Mark Seeman implementing the «fizzbuzz» program in three languages: C#, Haskell, and then Clojure. Note that he’s using a REPL to do the Clojure implementation, and it sure seems like he’s having more fun doing it than in the other languages! (He’s using a mostly abandoned editor called LightTable, which was groundbreaking when it was released.)

And here’s a video of the aforementioned and amazing Mike Nygard showing how to do TDD using Clojure and the Datomic database, from inside Emacs. (Many people equate Clojure with Emacs — but I think only 50% of people in the Clojure community use Emacs. I put this video last to demonstrate how many options viable Clojure editor options there are!)

The Amazing Clojure Community, Parting Thoughts, and What I’d Like To Write About In The Future

I’ve found the Clojure community to be incredibly amazing and supportive. There’s a Clojurian Slack Channel that is incredible. The Clojure/conj and other conferences tend to attract very experienced developers who share similar sensibilities. All these conferences tend to invite some of the most accomplished researchers and contributors in the programming domain.

An example: one of most amazing talks I’ve seen was from the 2016 Clojure/conj where Brian Goetz, architect for the Java language, talked about his stewardship of the Java ecosystem. What’s unmistakable and so admirable is his sense of responsibility to not break code that nine million developers have written.

Here’s a talk from Tiago Luchini on writing declarative domain models-after you watch this, you’ll never want to deal with a SQL or noSQL database again… and which is why I switched to using the Datomic database, from the same organization that brings us Clojure.

Some Great Learning Aids for Clojure:

I recommend the following books to get up the speed on Clojure:

(I’ve bought almost every book on Clojure that I could find. I like every one one of them.)

(I also want to thank Mike Fikes for all his help. I’ll never forget how, in a moment of impulse a couple of years ago, I asked whether he’d be available to chat, to teach me about the difference between the different ClojureScript runtimes, and how to connect them to IntelliJ. You’re amazing, and keep up the great work! Sorry, and thank you!!!)

The work within the Clojure community is so interesting, I subscribe to the Planet Clojure RSS feed, which aggregates articles about Clojure and functional programming from many sources: http://planet.clojure.in/.

I have a lot of enthusiasm to write in the future about the following because it illuminates the First Ideal of Simplicity and Locality, and the Second Ideal of Focus, Flow, and Joy.

• The Clojure programs I wrote to analyze writing patterns throughout development of The Unicorn Project (download the excerpts).
• My top twenty videos from the Clojure community that blew my mind.
• My adventures in trying to learn static functional programming languages, like Haskell, and learning category theory-that’s been unexpectedly rewarding. This is the world of monoids, functors, applicative functors, and the monad.
• Doing a screencast of analyzing the Stack Overflow 2019 Survey data to better understand the demographics of Clojure developers, and why they rank as the highest paid population.

Footnotes

1. Thank you to Parker Matthew from Pivotal for cementing this connection for me in one of our many amazing conversations. It’s interesting to note the amazing tension and mutually supportive nature between Dr. Csikszentmilhali’s work, and the work of Dr. Anders Ericsson about deliberate practice, described in . The dueling dynamic of «practice as a transcendental experience» vs. «practice as hard work and perseverance» is magnificently described in Dr. Angela Duckworth’s  book «Grit: The Power of Passion and Perseverance»↩
2. The person who created the Strange Loop conference is Alex Miller, who is also a very prominent figure in the Clojure community. In another strange twist of history, Adrian Cockcroft (@adrianco) told me that this was his favorite conference in 2014. I filed it away at the time. Years later in 2017, I found myself devouring video after video from this conference — it is a magnificent program, a confluence of Papers We Love, static types, category theorists, hackers working on amazing projects, and even DevOpsy things. I’ve been wanting to attend for the last three years, but had to cancel at the last minute each time. 2020 will be the year, I’m sure!!! 

Further Reading

• Five Invaluable Laws of Growth for Scrum Masters.
• Clojure: The Perfect Language to Expand Your Brain?.
• Building a Small Bank With Clojure The Clojure Programming World: Libraries and Other Tools.