A validation DSL in Go, for the API Gateway that opened us to partners

In the previous post I wrote that 143 scenarios guard the gateway the job boards and partners call, and that it deserved a post of its own. This is that post.

The résumé service was an internal service. It stored every candidate profile of the job boards, it trusted its callers, and until the summer of 2019 its callers deserved it : they were ours. Then the job boards themselves, and a partner, needed to write into it. Different teams, different release cycles, different ideas of what a phone number looks like. An internal service that trusts the outside world does not stay healthy for long.

So in July 2019 we put a gateway in front of it, with three jobs : authentication, authorization, validation. The first two took a few days. Validation took five weeks — and then two days, once I stopped writing it by hand. The refactor that replaced it deleted 1,799 lines 🚀

The gateway is also what let us open the résumé store to the outside without touching the store. The service kept trusting whatever reached it, because the only thing that could reach it was the gateway. A caller got a token whose audience is its own name and whose roles say which routes it may call ; the validation made sure a job board or a partner could only write profiles in its own name. Three concerns, one door, and behind it a service that never learned there was an outside 🔍

A gateway that says no

The gateway is a small Go service on Gin. Callers get a JWT signed with RS256 ; the token carries who they are and what they may do, as a list of roles. Every route declares the one role it requires, and the table of routes reads like the access policy it is :

// read : checked, then forwarded untouched
r.GET("/accounts/:id", jwt, limiter, HasRole("account-read"), Proxify(services))
r.GET("/accounts/:id/attachment", jwt, limiter, HasRole("attachment-read"), Proxify(services))

// write : checked, validated, then forwarded untouched
r.POST("/accounts", jwt, limiter, HasRole("account-create"), CreateAccount(services))
r.PATCH("/accounts/:id", jwt, limiter, HasRole("account-update"), PatchAccount(services))
r.DELETE("/accounts/:id", jwt, limiter, HasRole("account-delete"), Proxify(services))

The check itself is a dozen lines of middleware :

func HasRole(requiredRole string) gin.HandlerFunc {
  return func(c *gin.Context) {
    for _, role := range c.GetStringSlice("CLAIMS_ROLES") {
      if role == "all" || role == requiredRole {
        c.Next()
        return
      }
    }
    c.AbortWithStatusJSON(http.StatusUnauthorized, gin.H{
      "status": http.StatusUnauthorized,
      "error":  fmt.Sprintf("This action require '%s' role", requiredRole),
    })
  }
}

Twenty-one routes in the autumn of 2019, forty-one at the end, most of them a pure proxy ; the rate limiter per caller came later. The ones that matter are the writes : a profile that enters the resume service is indexed, enriched, searched by recruiters and replayed for years. A bad date or a salary of minus one does not fail loudly there — it fails quietly, downstream, three weeks later. The gateway’s job is to say no at the door, with a message precise enough that the caller fixes the payload without calling us.

Five weeks of if x == nil

The first version did what every Go tutorial does : a struct per JSON object, pointer fields so that nil means “absent”, unmarshal, then check. One commit per field, for five weeks. It looked like this — condensed — for seven hundred lines :

func (a *Account) ValidateCreate(data []byte) error {

  err := jsonapi.Unmarshal(data, a)
  if err != nil {
    return errors.New(fmt.Sprintf("invalid json: %s", err.Error()))
  }

  if a.Profile == nil {
    return errors.New("field profile required")
  }

  if a.Profile.Supplier == nil {
    return errors.New("field profile.supplier required")
  }

  if a.Profile.Supplier.Name == nil {
    return errors.New("field profile.supplier.name required")
  }

  supplierName := strings.ToLower(*a.Profile.Supplier.Name)
  if !contains(suppliers, supplierName) {
    return errors.New("invalid profile.supplier.name")
  }

  // ... 700 more lines
}

Three things were wrong with it, and none of them was the typing.

The path is written twice. Once as Go field access, once as a string in the error message. They drift : a renamed field keeps its old name in the error, and the caller hunts for a field that does not exist.

Every rule is three rules. Present ? Not null ? Valid ? The interesting part — valid — is buried under the two boring ones, and a reviewer cannot see the policy for the plumbing.

PATCH doubles everything. On a create, a missing required field is an error. On a patch, a missing field means “leave it alone” — but if it is there, every other rule applies. The hand-written version of ValidatePatch said it best :

func (a *Account) ValidatePatch(data []byte) error {
  // ...
  //TODO: validate...
  return nil
}

After five weeks, the update route validated nothing. Writing the same seven hundred lines a second time, with every required flipped, was not going to happen 😅

The same rules, as sentences

What I wanted to write was the policy itself, one sentence per field, readable by the people who own the contract. This is the rule set, condensed :

func validate(c *gin.Context, json Field) {

  // a caller can only write profiles in its own name :
  // the payload must match the token
  json.field("profile.supplier.name").
    required().string().equal(c.GetString("CLAIMS_AUDIENCE"))

  json.field("profile.supplier.candidateId").
    required().string().notBlank()

  json.field("profile.lastActivityDate").
    required().string().notBlank().timeRFC3339()

  contact := json.field("profile.resume.contactInfo").required()

  contact.field("internetEmailAddress").
    required().string().notBlank().email()

  contact.field("birthDate").
    optional().string().notBlank().number().yearBetweenNowAnd(1900)

  postalAddress := contact.field("postalAddress").optional()

  postalAddress.field("postalCode").
    required().string().notBlank()

  postalAddress.field("countryCode").
    required().string().notBlank().toLower().countryCode()

  languages := json.field("profile.resume.languages").optional().array()

  for _, language := range languages {
    languageCode := language.field("languageCode").required()
    languageCode.field("id").required().string().toLower().languageCode()
    languageCode.field("label").optional().string().notBlank()
  }
}

The first version was ten functions, 150 lines of Go on top of gjson, which reads a path in a JSON document without unmarshalling it. No struct, no reflection, no tags. Four ideas carry it.

Idea 1 : the compiler checks the order of the rules

A rule chain moves through types. field() returns a Field. .string() checks the JSON type and returns an SField, which is where notBlank(), email() or oneOf() live. .number() parses and returns an IField, the only place where between() exists :

type Field struct {
  patch  bool
  skip   bool
  path   string
  result gjson.Result
}

type SField struct{ Field }             // a JSON string
type IField struct{ value int; Field }  // ... parsed as an integer
type TField struct{ value time.Time; Field }

func (f Field) string() SField {
  s := SField{Field: f}
  if f.skip {
    return s
  }
  if f.result.Type.String() != "String" {
    panic(fmt.Sprintf("field %s is not a string", f.path))
  }
  return s
}

func (f IField) between(min int, max int) IField {
  if f.skip {
    return f
  }
  if f.value < min || f.value > max {
    panic(fmt.Sprintf(
      "invalid field %s value, must be between %d and %d",
      f.path, min, max))
  }
  return f
}

json.field("x").between(1, 12) does not compile. Neither does .email() on a number. The nonsense rules are caught by go build, not by a test somebody forgot to write.

Idea 2 : absence is a state, not an error

Every field carries a skip flag, and every verb starts with if f.skip { return f }. optional() sets it when the field is absent or null, and from there the rest of the chain — and every field nested under it — is a no-op :

func (f Field) optional() Field {
  f.skip = !f.result.Exists() || f.result.Value() == nil
  return f
}

That is what makes postalAddress.field("postalCode").required() read correctly : the postal code is required if there is a postal address. The hand-written version needed a nested if for that sentence. Here it is the default.

Idea 3 : one rule set for POST and PATCH

The flag that paid for the whole DSL is the other one, patch. required() reads it :

func (f Field) required() Field {
  if f.skip {
    return f
  }
  if !f.patch && !f.result.Exists() {
    panic(fmt.Sprintf("field %s is required", f.path))
  }
  if f.result.Exists() && f.result.Value() == nil {
    panic(fmt.Sprintf("field %s can't be set to null", f.path))
  }
  if f.patch {
    f.skip = !f.result.Exists() // absent on a patch : leave it alone
  }
  return f
}

On a create, a missing required field panics. On a patch, it quietly becomes a skipped field — but a field that is present goes through the full chain, and an explicit null on a required field is refused in both cases. The two entry points differ by one call :

// POST /accounts
validate(c, parse(data).prefix("data.attributes"))

// PATCH /accounts/:id
validate(c, parse(data).forPatch().prefix("data.attributes"))

The //TODO: validate... of the update route was closed by forPatch(). One policy, two verbs, no drift between them.

Idea 4 : rules return values

A chain does not only check ; it ends on a typed value, and the next rule can use it. Cross-field rules stay one-liners :

salary := jobSearch.field("salary").optional()
salaryMin := salary.field("min").required().string().number().min(0)
salary.field("max").required().string().number().min(0).min(salaryMin.value)

startDate := employerOrg.field("startDate").required()
yearMonth := startDate.field("yearMonth").required().string().yearMonth()
startYear := yearMonth.firstChars(4).number().yearBetweenNowAnd(1900).value
startMonth := yearMonth.afterChars(5).number().between(1, 12).value

And a verb is just a function, so a rule can ask another service. Job categories, industries and education levels are codes owned by a referential ; the DSL checks them where the other rules are, not in a second pass :

jobCategory.field("id").required().string().
  validClassificationId(services, supplier, "jobCategories")

The error names the field

Paths compose. field("profile") then .field("supplier") builds profile.supplier ; array() appends the index. The caller gets the exact address of the problem, in the vocabulary of their own payload :

{
  "status": 400,
  "error": "invalid field profile.resume.employmentHistory.employerOrgs[0].startDate.yearMonth value, must be a year between 1900 and now"
}

The path is written once, in the rule. It cannot drift from the message, because it is the message.

The part that is not idiomatic

Every verb fails with panic, and there is exactly one recover in front of the rule set :

func ValidateCreate(data []byte) (validationError error) {
  defer func() {
    if r := recover(); r != nil {
      validationError = errors.New(fmt.Sprint(r))
    }
  }()
  // ...
}

Go people will wince, and they have a point : panic is not an error-handling mechanism. I kept it, knowingly, for one reason — a chain. a().b().c() cannot return (Field, error) at each step ; the alternatives are an error accumulated inside the field and checked at the end, or a closure per rule. Both put plumbing back in the sentences I had just cleaned. The panic never leaves the package, the boundary is two functions wide, and the package’s public API returns a plain error.

The real limit is elsewhere : it stops at the first error. A payload with four problems takes four round trips to fix. For callers that are machines, integrated once by an engineer reading the message, that was fine for four years. For a form with a human in front of it, it would not be — and collecting errors instead of panicking is the change I would make first.

The contract is a feature file

The gateway ships 143 Gherkin scenarios, 123 of them on the create route alone, run by godog against the real binary :

570 steps in 614 milliseconds. A suite that runs in less than a second is a suite that gets run. Each rule has its refusal written down :

Scenario: no profile.supplier field
  Given a running api
  When I send a "POST" request to "/accounts" with json:
    """
    { "data": { "type": "account", "attributes": { "profile": {} } } }
    """
  Then the response code should be 400
  And the response should match json:
    """
    { "status": 400, "error": "field profile.supplier is required" }
    """

Most of those scenarios were written against the hand-written version, one per commit, one per field. The refactor changed the engine under them and had to keep them green — which is the only reason I dared delete seven hundred lines of validation in two days.

The numbers

• 1,799 lines deleted, 560 added, in the refactor that moved the rules to the DSL.
• The account model went from 930 lines to 218.
• 41 methods in the DSL at the end ; the first version had ten functions, in 150 lines.
• 143 scenarios, 41 routes, one role per route.
• 0 lines of validation on the update route before ; the full rule set after, for one call to forPatch().

It kept evolving

A DSL is finished the day somebody else adds a verb. Six weeks after the refactor a teammate needed to cap free-text fields, so that a pasted novel does not end up in a job title :

func (f SField) checkOverflow() SField {
  return f.maxLength(5000)
}

Three lines, no question asked, and the policy changed where the policy is written. Others followed as the contract grew : department and region codes a few weeks later, a suffix check, a range of years, then a forceRequired() and a link() in the years after. The gateway grew around the rules too — a rate limiter per caller, a refresh endpoint, twice the routes — and when the service behind it was rewritten from Scala to Go in 2021, the gateway’s part of the move was a few lines of configuration. Callers never noticed.

None of that was mine. That is the test a DSL has to pass : not whether its author finds it elegant, but whether the next person extends it without reading the engine ✅

Lessons learned

• Validate at the door, forward untouched. The gateway reads the payload with gjson and never re-serialises it ; what was validated is byte for byte what the service receives.
• Write the path once. A rule that carries its own path cannot lie in its error message.
• Make absence a state. skip turned every nested if into the default behaviour, and forPatch() turned two policies into one.
• Let the types order the rules. If .between() only exists on a number, nobody has to test that it is never called on a string.
• Own the unidiomatic part. A panic behind a two-function boundary bought readable rules ; stopping at the first error is the price, and I know where I would pay it back.

The résumé service still trusts its callers. It can afford to : there is someone at the door who does not 😉