How a JVM shop became a Go shop

My mission at the classifieds group ended last month, six years and eight months after it started. When I arrived, the services around the job boards were Java and Scala applications on hosts in a datacenter. Today the part of the platform I worked on is 25 Go services on Google Kubernetes Engine — the teams around it run more — each with its own Helm chart and its own pipeline. And 14 of them, with three Cloud Functions and a Vue front, are the CV database : the recruiter product, the group’s second source of revenue, rebuilt from a ten-year-old application that nobody dared to touch. The JVM survives where it belongs, in the Beam jobs of the data platform.

None of that was planned. It started with a hello world that would not boot.

250 MB for Java

Spring 2017. The data platform was three months old and the next thing to move to the cloud was the tracker, the service that receives every event from the job boards. The one we had was an old Scala / Play application, with years of dead code in it. Before porting anything, we put a Scala Play hello world on the new Kubernetes cluster, in a pod with 250 MB of memory. OOMKilled before the first request.

The ops engineer’s position was reasonable and final : 250 MB of RAM is what a pod gets. A JVM wants a heap sized in advance, a warm-up before it is fast and a base image with a runtime in it ; a cluster wants small requests, fast restarts and many replicas. For a service that parses JSON and publishes it to Pub/Sub, those two wish lists did not overlap.

So I proposed Go to the CTO. Nobody on the team had written a line of it, which is why I proposed one free freelance day to find out whether it could carry the platform’s most exposed service. Hit or miss.

One evening, under a millisecond

The first commit is dated May 4, 2017, 17:20. Nine more followed that evening : the Dockerfile, the pipeline, the Pub/Sub publisher — and one written by the pipeline itself, bump to 0.1.0. The whole build and packaging story still fits in seven lines :

build: deps
  CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o target/app .

FROM scratch
ENV GIN_MODE release

ADD target/app /
CMD ["/app"]

A static binary in an empty image. No runtime, no shell, no package manager — the container is the process. And this is its access log under a bench :

Nine requests, nine 200, between 120 and 503 microseconds. Not milliseconds. The hot path is short on purpose — validate, pick the topic, publish, answer — and the publish never makes the caller wait :

func Publish(topic *pubsub.Topic, msg []byte, attrs map[string]string) {
  ctx := context.Background()
  result := topic.Publish(ctx, &pubsub.Message{
    Data:       msg,
    Attributes: attrs,
  })

  // the HTTP response is already gone : the outcome is only logged
  go func(ready <-chan struct{}, result *pubsub.PublishResult) {
    <-ready
    if id, err := result.Get(ctx); err != nil {
      log.Printf("Failed to publish in Pubsub: %v -- %v", id, err)
    }
  }(result.Ready(), result)
}

HTTPS came twelve days later, horizontal autoscaling on May 29, a Gatling job in the pipeline in November. The production values still say the same thing today : two pods at rest, ten at most. The customer loved what the tracker did on the cluster. After that, the question “which language for the next service ?” was not asked again 🚀

The shape of a service

One evening proves a language can boot. The next six years are about whether a team can own it. What made that possible is that the tracker became a blueprint : every Go service we run has its shape, cloned from it into the next service, then into all the others, tests included :

statistics/
├── Dockerfile               FROM gcr.io/distroless/base
├── Dockerfile-dev           the hot-reload image (air)
├── Makefile
├── go.mod
├── main.go                  four lines : app.New().Run()
├── app/                     wiring : config, logger, router, clients
├── handlers/                one file per resource
├── models/
├── services/                Elasticsearch, Pub/Sub, accounts
├── router/                  Gin routes + middlewares
├── bdd/
│   ├── features/*.feature   the contract, in Gherkin
│   ├── hooks_test.go
│   └── run_test.go          godog runner
└── scripts/
    ├── cloud-build/         build · bdd · deploy · promote · smoke
    ├── helm/<service>/      Chart.yaml · deployment · service · hpa
    └── jenkins/             Jenkinsfile · JenkinsfileProd

The dependency list is as short as the tree, and it is the same list in almost every go.mod we have :

module github.com/acme/statistics

go 1.13

require (
  github.com/cucumber/godog v0.8.0          // BDD : .feature files are the tests
  github.com/gin-gonic/gin v1.7.7           // HTTP
  github.com/olivere/elastic/v7 v7.0.31
  github.com/spf13/viper v1.6.2             // config, 12-factor style
  github.com/tidwall/gjson v1.12.1          // JSON without a struct per payload
  go.uber.org/zap v1.13.0                   // structured logs
  gopkg.in/DataDog/dd-trace-go.v1 v1.20.1
)

Gin for HTTP, viper for configuration, zap for logs, gorm or the Elasticsearch client for storage, godog for the tests. No framework debates for six years. An engineer who has shipped one of these services can open any of the others and know where everything is. That is worth more than any individual library choice.

Go people will notice that this is not how a Go project usually looks. No cmd/, no internal/, no flat package named after what it does — a package per layer instead, handlers, models, services, router, the way a Spring or a Play application is laid out. That is on purpose. The team came from the JVM, and a layout they already knew was one less thing to learn on the day they learned a language. Idiomatic Go was never the goal ; a team that could clone a service on Monday and ship one on Tuesday was. The blueprint came with everything but the business : the tests wired in — a bdd/ folder with a running api step, godog in the pipeline — and the pipeline itself, build, deploy, promote, smoke. Later my teammates reworked it into a proper template, runnable as cloned, which is the fate every blueprint should hope for.

A product nobody dared to touch

The CV database is where recruiters search résumés, save their searches, get alerts and spend credits to contact candidates. It is the group’s second source of revenue. In 2018 it was also ten years old : PHP, a JavaScript framework that had stopped being maintained, and a build chain that only ran inside a development VM kept alive for that purpose. Product was scared to refactor it, and just as scared to start a version 2 : nobody wants to be the one who breaks the product that pays the bills.

We did it. Three of us wrote 96% of it.

The rule was the one the data platform had already used twice : make the new path live first, then drain the old one. No rewrite in place, no freeze, no cutover night. A new product, built next to the old one, that takes the recruiters when it is ready for them.

Fourteen services, three functions and a front

Each service is small enough to be understood in an afternoon. The whole product starts on a laptop with one script : one compose file per service, side by side in the same folder.

I started the front too. Vue CLI on October 5, 2018, then the unglamorous part — the pipeline, the CDN upload, and end-to-end tests running in a container against the deployed application, with page objects and a screenshot on every failure. A front that is tested against production on Christmas Eve is a front a back-end team can keep shipping 😅

The gateway : configuration, not code

Fourteen services need one public entry point. We started ours in December 2018 as a proof of concept on KrakenD, a Go API gateway configured with JSON, and it never left. Today it is 139 endpoints in 35 templates, and an endpoint looks like this :

{
  "endpoint": "/credits/pre-check",
  "method": "POST",
  "output_encoding": "no-op",
  "timeout": "10s",
  "extra_config": {
    "auth/validator": {
      {{ template "config_jwt_user_validator.tmpl" . }},
      "audience": ["recruiter"]
    },
    "acme-gateway/authenticator": {}
  },
  {{ template "config_headers_to_pass.tmpl" . }},
  "backend": [
    {
      "url_pattern": "/pre-check",
      "encoding": "no-op",
      "host": ["{{ .values.credits.host }}:{{ .values.credits.port }}"]
    }
  ]
}

Who may call, which headers travel, where it goes. The Go in the repository is a thin layer of plugins ; the rest is templates. Ten people have committed to it : routing as configuration is the kind of code people dare to touch.

Credits : where the money is

A recruiter pays with credits, so the credits service is the one that must never be wrong in the customer’s disfavour. Its price list is a file :

{
  "resume_view":     { "default": 1 },
  "resume_download": { "default": 1 },
  "resume_contact":  { "default": 0, "bulk": 1 },
  "resume_forward":  { "default": 1 },
  "resume_print":    { "default": 1 }
}

And the handler is written around one question : what happens if we fail halfway ? Condensed :

for _, action := range request.Actions {
  cost, err := accounts.GetCost(action.Action, action.Origin)
  // ...
  already, _ := credits.GetLastAction(ctx, action, request)

  // every action is recorded under the request's correlation id
  if _, err := credits.InsertAction(ctx, action, request); err != nil {
    credits.DeleteAllActions(ctx, request.CorrelationId)
    // 500
  }

  // a résumé already paid for is free the second time
  if credits.LightConsumption && already != nil {
    continue
  }
  total += *cost
}

resp, err := accounts.Debit(c, token, request.CorrelationId, total)
if err != nil || resp.IsError() {
  // the debit failed : forget everything this request recorded
  credits.DeleteAllActions(ctx, request.CorrelationId)
  c.JSON(http.StatusPaymentRequired, gin.H{
    "status": http.StatusPaymentRequired,
    "cost":   total,
    "error":  "Not enough credits",
  })
  return
}

No distributed transaction, no saga framework. A correlation id, a compensation, and HTTP 402, the status code everybody knows and nobody uses. Two more routes, /pre-check and /check, answer the question before anything is spent.

Alerts : one function per job

Saved searches send emails, morning or afternoon, on the days the recruiter picked. The first version was one Cloud Function doing everything. On March 2, 2020 it became a fan-out : a trigger that reads the alerts due now and publishes one message per alert, and a worker that runs one search and sends one email.

// On instance cold start
func init() {
  db, err = sql.Open("postgres", dsn)
  // ...
  // Only allow 1 connection to the database to avoid overloading it
  db.SetMaxIdleConns(1)
  db.SetMaxOpenConns(1)
}

func TriggerEmailAlerts(ctx context.Context, msg PubSubMessage) error {
  rows, err := db.Query(selectAlerts+" FROM searches WHERE batch_id=$1", batchId)
  // ...
  for rows.Next() {
    // one saved search = one message = one execution of the worker
    result := processTopic.Publish(ctx, &pubsub.Message{Data: jsonBody})
    if _, err = result.Get(ctx); err != nil {
      inError++
      continue
    }
    toProcess++
  }
  // ...
}

A slow search delays one email, not the batch, and the platform scales the workers, not us. A third function takes over when an alert finds nothing : it widens the search, keeps the first three variants that return résumés, and sends those instead.

New path live first, then drain the old one

• December 2018 — front, gateway, authentication and users deploy to production, with end-to-end tests running against it.
• October 14, 2019 — a token bridge between the two versions : one login opens both.
• November 26, 2019 — the version 2 opens as a beta, next to the legacy, with a banner saying so.
• June 3, 2020 — the commit message says it all : “change beta to new”. 190 days of beta, and the busiest month of the whole product : 413 commits.
• February 2021 — the importer : accounts, saved searches and alerts of the legacy, converted and pushed through the gateway like any other client.
• September 15, 2022 — the importer is removed. There is nothing left to import.

The importer deserves a word, because it is where a migration is usually dishonest. Ten years of saved searches do not map cleanly onto a new search engine : regions the new search does not know, boolean queries the new parser rejects. Every converter has its tests, and every conversion that loses something says so. Condensed, messages translated :

func addLossesRegionWish(row models.UntypedSQLRow, losses []string) []string {
  if row["region_wish"] == nil {
    return losses
  }
  wishes := strings.Split(row["region_wish"].(string), "|")
  values := funk.SubtractString(wishes, []string{""})

  // the regions the new search does not know
  kept := funk.SubtractString(values, unsupportedRegions)

  // nothing usable left : tell the recruiter which criteria were dropped
  if len(values) > 0 && len(kept) == 0 {
    losses = append(losses,
      fmt.Sprintf("Wished region: %s", strings.Join(values, ", ")))
  }
  return losses
}

The recruiter receives an email listing what was imported and what could not be. A customer who is told what changed trusts the new product. A customer who discovers it does not.

A migration you can explain to the customer is a migration you can finish.

The second Scala service

The résumé API, the service behind the gateway that job boards and partners call, was still a Scala / Akka application. Its replacement in Go started on January 8, 2021 and was being prepared for production on March 17 : ten weeks. It kept the old health-check path, so that nothing around it had to change on the day it took over.

The first commit of that repository is not mine. By then Go was not my proposal anymore, it was how the team wrote services.

The feature file is the contract, where it matters

Most services are born with a bdd/ folder and the template’s scenarios — health, unknown route — run by godog against the real binary in a pipeline step of their own. For nine services that is all there is, and the search service and the gateway have none. I will not pretend otherwise : they sit behind the gateway, and the front’s end-to-end tests are what exercises them.

Where a service is a contract with somebody else, the feature files are the specification : 143 scenarios on the gateway the job boards and partners call, 72 on the tracker, 32 on the events API, 11 on authentication. 291 in total. That gateway deserves a post of its own. Two of the tracker’s scenarios, sanitized :

Feature: Test health endpoint

  Scenario: Health-Check
    Given a running tracker
    When I send a "GET" request to "/health"
    Then the response code should be 200
    And the response should match json:
      """
      { "status": "UP" }
      """

Feature: Test contract events

  Scenario: Accept a valid contract event from a known source
    Given a running tracker
    When I send a "POST" request to "/contract" with json:
      """
      {
        "uuid": "request-0001",
        "group": "career",
        "domainType": "contract",
        "action": "receive_contract",
        "contractId": "contract-0042",
        "date": 1427380852000,
        "ip": "192.168.0.1",
        "physicalServerOrigin": "tracker.front.local",
        "source": "tracker"
      }
      """
    Then the response code should be 201
    And the response should match json:
      """
      {
        "uuid": "request-0001",
        "group": "career",
        "domainType": "contract",
        "id": "00000000-0000-5000-8000-000000000000",
        "action": "receive_contract",
        "entity": "contract-0042",
        "date": "2015-03-26T14:40:52Z",
        "ip": "192.168.0.1",
        "physicalServerOrigin": "tracker.front.local",
        "source": "tracker"
      }
      """

The second one says more than it seems to : a date that arrives as epoch milliseconds leaves as RFC 3339, the caller’s contractId becomes the platform’s entity, and the event gets an id of its own — the one the whole data platform deduplicates on.

A team that integrates the tracker reads this file, not our code. The rule I would keep : write the scenarios where a stranger depends on you, and do not decorate the rest.

Four generations of CI, one scripts/ folder

We also went through four CI systems, and the services barely noticed. Concourse in early 2017, running the tracker’s pipeline on day one. Codefresh at the end of that year. Jenkins and Cloud Build together in the summer of 2019, Cloud Build alone by the summer of 2020. Each service keeps its pipeline definitions under its own scripts/ folder, one directory per tool. Moving CI was adding a directory, not rewriting a service.

Where the Scala went

The chart below counts, per year, the Go and Scala files we touched. It is a volume proxy, not lines of code, but the shape is unambiguous : Go passed Scala in 2018 and never looked back, and Scala’s tail is the Beam jobs — the one place where the JVM was the right tool and stayed.

One counter-example, kept because it is true : a PDF exporter started in Go in May 2022 and became a Node service fifteen days later. Rendering HTML to PDF is a browser’s job, the tooling for it lives in Node, and the language followed the tool. The rule was never “Go everywhere” ; it was “the smallest thing the pod can afford” 🙃

Whose platform it is

In 2017 I wrote 98% of the commits on the Go repositories. In 2023, 18%. Over the same years the people committing to the platform went from 12 to 53 at the peak, in 2022.

The numbers I like best are the ones I am absent from. The Scala résumé API was replaced by a repository I did not start. The search engine moved from Elasticsearch to OpenSearch last winter, service by service, in pull requests I only read. And in September 2022 a teammate retired the legacy importer from the gateway, the saved-search service and the bot, in one day, because there was nothing left to import. The product had outlived its own migration, and nobody needed me for the last step.

Prove it in production, then let go of it.

Lessons learned

• A working service beats a slide deck. A tracker answering in microseconds settled a language debate that a document never would have.
• Pick the language the pod can afford. The constraint was memory and restart time, not elegance ; Go won on the pod’s terms, and the team’s terms followed.
• Same shape, every service — and a shape the team already knew. Twenty-five services one engineer can navigate blind are worth more than twenty-five idiomatic ones.
• Rebuild next to the legacy, never inside it. A beta, a token bridge, an importer that admits its losses : the old product was drained, not switched off.
• Write scenarios where a stranger depends on you. 143 on the public gateway, 3 on an internal service, and no shame about the difference.

Somewhere on that cluster, a Dockerfile shaped like the one from May 2017 is still the first file of the next service — and I am no longer there to see it 😉