Hat tip to Stuart Halloway… once again a 10 minute conversation with Stu grew into a combination of code and writing that helped me clarify my thoughts.

I've been working on new content for my Monolith to Microservices workshop. As the name implies, we start with a monolith. Everyone gets their own fully operational production environment, running a fork of the code for Lobsters. It's a link sharing site with a small but active group of users. I chose that application for a few reasons:

  1. It's very well written and has good internal structure. That makes it malleable enough to use in a classroom setting.
  2. It's small enough to be useful during a week-long workshop.
  3. The domain is familiar enough that students don't need a ton of domain-specific introduction.

One of the features of Lobsters is "hats." From the site's own description:

Hats are a more formal process of allowing users to post comments while "wearing such and such hat" to give their words more authority (such as an employee speaking for the company, or an open source developer speaking for the project).

Hats are the first feature that we factor out into its own microservice. I thought it might be interesting to walk through that process and how the new service is defined.

This is going to be a long post, because I'm trying to recapitulate my whole thought process. Please let me know if I've skipped steps.

Point of Departure

The feature basically works like this:

  • A user can have zero or more "hats." Each hat has a short name that designates a project or product. Examples include "bsd" or "docker".
  • When posting a comment or sending a private message, the user can choose a hat to "wear". This could be to demonstrate credibility or make an official statement.
  • It's the job of site admins to verify the user's identity and standing relative to the hat.

Given that description, it's pretty natural to think about an API like this one. (Follow the link to read an API doc in "Blueprint" format.)

As you read the API description, notice that most of the routes read like "create this thing", "delete this thing", and so on. It sounds suspiciously close to CRUD, and that should trigger an uneasy memory about entity services.

Entity services are what you get when you only think about the data and not how you are going to use it.

A more subtle problem with this API is that it provides the wrong point of entry for the most common use case. When a user starts posting a comment, the site needs to find out what hats that user wears. It seldom needs to find all the wearers for a hat.

We can plaster over this gap by adding more routes to the API. But there's probably a better way to approach the whole issue.

Think About Behavior

If I have a mantra for architecture and design, it's "Think about the behavior." I advise people to evaluate their designs by walking through use cases. What components have the ability to make progress toward the goal? How does the flow of control get there? What information do I need to supply to it?

If we think about behavior in the "hats" feature, we'll see that the original API has some big gaps.

  • How does an admin know that someone needs a hat?
  • When does the system need to read all the users for a hat?
  • Can someone who has a hat bestow it on someone else?
  • What happens if someone has a hat when they make a comment but later loses that hat?

Let's take a behavior-oriented view on the hats. In particular, let's think about the lifecycle:

  1. A user requests an admin to bestow a hat upon them.
  2. The admin can grant that request. In that case, the hat is attached to the user.
  3. The admin can reject that request.

If we stick with the CRUD style API, that behavior is "pushed" out to someplace else. Requests, approvals, rejections all have to go in the caller. What's left in the service isn't enough to be interesting. We'd have a caller that still knows all the details of the data. Any other callers of the Hats service would also be completely coupled to the details of the data. It might as well just be an RDBMS table.

Why would we just take a single table from an RDBMS and put it on the far end of an HTTP interface?

Take Two

Let's try making an API that maps the actions in the original controller. After all, we're decomposing a monolith into microservices. The monolith already works so we know the current design solves for the features needed.

The controller has these methods:

  • index
  • build_request
  • create_request
  • requests_index
  • approve_request
  • reject_request

Notice something interesting about those methods? None of them talk about hats! The only one that actually cares about hats is index, and all it does it get all the hats to serve the hats page that shows everything. That's the least interesting of the behaviors. The hats controller appears to be almost entirely about the process of requesting a hat and approving or rejecting such a request. Let's set a bookmark called "Requests" here and come back to it later.

Where do hats actually get used? Let's look at comments. When a user starts to post a comment or reply to another comment, there's this bit of code that checks to see if the user has any hats. If so, the comment fragment offers the user the option to "put on a hat". That gets carried through to the comments controller, where the hat is attached to the comment. (Now we know what happens if the user doffs a hat… old comments still show that they did wear the hat at the time of the comment. Nice!)

The comments controller doesn't have any methods about the hats, although it does use hat data. It gets the hat data from the User model which has_many :hats.

Now we understand the feature much better. Instead of talking about it in the abstract, we can talk about when each part of the feature is activated and used. We understand the lifecycle of the data: who creates it? When? In response to what signals?

All of these are essential to designing successful microservices! If you try to design services in a vaccuum, you'll find they don't work together and you need a bunch of glue code in the service consumers. (Hint: if you start trying to solve distributed two-phase commit across microservices, then you haven't gotten concrete enough about the actual use cases.)

Not One But Two

Recall that the HatsController class seemed to be about requesting a hat more than the actual hat? Suppose we created microservice methods like:

  • Request hat
  • Approve hat request
  • Reject hat request
  • List pending hat requests
  • See my hat request

That would pretty much map one-to-one with the existing HatsController methods. Seems like an easy way to solve the problem. The only problem is that it seems a touch too specific. We can often make microservices simpler, more general, and more useful by abstracting the interface up one step. Let's try that out:

  • Request "thing"
  • Approve request for "thing"
  • Reject request for "thing"
  • List pending requests
  • See my request

"Request thing" and "Approve or reject request" seem to be pretty general ideas. I bet you can think of half a dozen other uses for that concept in your company. What is the "thing" though? We need some kind of concrete representation, right? Let's try to avoid premature commitment to that. I want to see how long we can just use a URL to identify the thing.

With this in mind, take a look at the new request service API. Like the HatsController, this doesn't say anything about hats. In fact, it seems to rely on external information for almost everything. It uses URLs to identify the person (or system) making the request, the thing being requested, and the person (or system) that approves or rejects the request.

This may seem like premature generalization and you may cry "YAGNI" at me. I understand. But there's something about YAGNI you have to keep in mind… it applies when you can keep the cost of change low and refactor across interfaces. Microservices do well at keeping the cost of change low, but are much more difficult when refactoring. The whole idea is that a service interface is isomorphic to a boundary in your organization. So we don't have collective code ownership, we don't have refactoring across the interface, and I contend that YAGNI must be greatly weakened as a rule.

What Was That About Hats Again?

Requests are sorted, more or less. Now we need to turn our attention to the question of the actual hats. We saw earlier that hats appear in three places:

  1. When building a page with comments on it, the (initially hidden) comment form needs to know what hats a user has.
  2. When posting a comment, copy the ID of whatever hat the user was wearing into the comment itself.
  3. When rendering a comment, display the text of whatever hat is attached to the comment.

Hats for a User

We can mostly handle the first case by querying for requests by subject (the subject being the user.) This could be done when the user logs in or the first time the user goes to a comment page.

However, the current method for querying by subject will return too much. First of all, it will return requests that are pending or were rejected. We can easily handle that using a matrix-query style of URL with both subject and status as parameters. Second, if we really do use the Request service for more than the hats feature, we don't want other "kinds" of requests appearing in the comments page. This one is trickier, since it needs a kind of meta-data that doesn't exist on the current definition of the Request service.

I'm not going to add that metadata just yet. That's because my workshop simulates the process of progressively splitting services out from a monolith. It's a common case to discover that your existing functionality is a subset of some more general, more valuable use case. That's when you go back and apply some data migrations and define a new API that deals with the general case. You then make the original API "magically" add the new metadata.

This may result in API names like "foo2" and "baz3." That's OK. The refactored, evolved version of your system won't look like a greenfield design would. Your system will show its history. Don't think of that as ugly scars. Think of it like laugh lines.

Adding a hat to a comment

When we find out what hats a user has, we get a list of URLs. Adding a hat to a comment doesn't need any additional interaction with requests or hats. Just copy the URL onto the comment where the code used to copy the ID.

Displaying a hat

One last interesting bit. We need to exchange a hat URL (from the 'object' of the original request) for a text label to display. This is the first thing we've encountered that is truly unique to hats… and it's basically a reference table.

This post is getting quite long as it is and I need to save something for people who come to my class. So I'll leave you with these quick thoughts:

  • Reference tables are a common need. Maybe we can create a more general service for curating reference tables. That would include information like who is allowed to add entries.
  • Someone may request a hat that doesn't exist yet. If the request is approved, then the hat "poofs" into existence. So what is the difference between "proposed" reference data and "current" reference data?
  • Is that lifecycle both general and interesting? Maybe there are two different APIs for dealing with curating the data versus just looking at current.
  • On the consuming side, we might decide to simply cache all the existing entries for a reference table. It's reasonable to have a query method on the table that says "give me the complete list of Hats" (or countries, or currencies, time zones, etc.) Fetching those at startup time is reasonable, but on a cache miss we still need to go ask about a single entry.
  • If we do create a reference data service, which deployment model do we want to use: A single instantiation of the service with all of our reference tables? Or one instantiation for each reference table? Think about the tradeoffs here both in terms of infrastructure cost and operations cost. (More instances = more infrastructure. Fewer instances = less ops cost at low volume, but more operations as scaling becomes harder.)

Conclusions

Our first idea is usually not the best one. To understand the boundaries and interface that make sense for a service, we have to think about it in situ. We aren't trying to model the world. We are trying to build systems that deliver features. Those features are specific and we must design our APIs to deliver those specific features. At the same time, however, we can often deliver the features just as easily by abstracting the API up one level. This makes a service more general and more reusable. It delivers more marginal value (i.e., it makes future work cheaper) and may even be simpler to write because it has less special-case logic or constraints.

We need to be careful to not push work into the gaps between services. One way to avoid that is to design APIs in terms of the caller's needs rather than the provider's view of the world.

Finally, sometimes the original service we set out to build evaporates completely when you discover that an apparently unitary concept is actually a composition of different concepts hiding under a noun.