Beyond unit tests

# Beyond unit tests

Integration and e2e tests in Go

![](images/633edce8ac2f63634c69ef590bc74108.svg)

Gopher: [https://github.com/egonelbre/gophers](https://github.com/egonelbre/gophers)

---

### Repository with example app and tests

![](images/2df0e65d2dae5691a7f6a31ebdaf5f79.png)

![](images/046a5ac1c00d1c39a16b375aa1e5e192.svg)

[https://github.com/antosdaniel/go-presentation-beyond-unit-tests](https://github.com/antosdaniel/go-presentation-beyond-unit-tests)

---

### What are we talking about today?

- When unit tests are not enough?
- Good patterns for different scenarios:
  - Testing with database
  - Testing APIs
  - Testing entire app/service
- How to keep tests easy to read, but not dreadful to write?

![](images/8581f28cd9a78d3e3b2a69cc65ac0033.gif)

---

### When unit tests are not enough?

When we want to:

- Test how components interact with each other.
- Make sure that setup is correct.
- Ensure that crucial paths of our app/service works under happy conditions.

![](images/d7784ed9b780ae5252a4430dbecd35eb.gif)

Note: Let's follow tittle of the presentation.

In the beginning of my career, I didn't really understand why I would want to test anything.

Integration tests feel more natural, but then you learn about unit tests. They are great, but they are not meant to solve everything!

---

![](images/969694331c35c0fd25371af1d65d72ee.png)

Source: [https://kentcdodds.com/blog/the-testing-trophy-and-testing-classifications](https://kentcdodds.com/blog/the-testing-trophy-and-testing-classifications)

Note: Testing trophy

---

### Testing with database

![](images/1d9270e9489bb3341e21de5a5d140800.png)

Note: You probably have some complex queries - you should have tests for them.

Back in the day, integration tests were nightmare not maintain. They were not portable between machines at all.

```yaml
  version: "3.9"
  services:
    db:
      image: "postgres:15.2-alpine"
      environment:
        POSTGRES_DB: expense_tracker
        POSTGRES_USER: postgres
        POSTGRES_PASSWORD: secret123
      healthcheck:
        test: [ "CMD-SHELL", "pg_isready -d $${POSTGRES_DB} -U $${POSTGRES_USER}" ]
        interval: 3s
        timeout: 60s
        retries: 10
        start_period: 5s
      ports:
        - "5432"
    migrate:
      image: "expense_tracker/migrate:latest"
      build:
        context: ../app_to_test/db
      environment:
        DB_URL: "postgres://postgres:secret123@db:5432/expense_tracker?sslmode=disable"
      depends_on:
        db:
          condition: service_healthy
```

```go
func StartDB(t *testing.T, ctx context.Context) DBContainer {
	// ...
	compose, err := tc.NewDockerCompose(path.Join(path.Dir(filename), "docker-compose-db-only.yaml"))
	// ...
	err = compose.WaitForService("migrate", wait.ForExit()).Up(ctx)
	// ...
	dbPort, err := dbContainer.MappedPort(ctx, "5432")
	// ...
	db, err := sql.Open("pgx", dsn)
	return DBContainer{ DB: db, DSN: dsn }
}
```

```go
func TestExpenseRepo_Add(t *testing.T) {
	db := StartDB(t, ctx).DB
	expenseRepo := api.NewExpenseRepo(db)

t.Run("successfully adds expense", func(t *testing.T) {
		expense := api.Expense{
			ID:       "c811c5d4-c38a-4f61-932d-d656c203b5f6",
			Amount:   123_50,
			Category: "food",
			Date:     time.Date(2020, 9, 5, 0, 0, 0, 0, time.UTC),
			Notes:    "some notes",
		}

err := expenseRepo.Add(expense)

require.NoError(t, err, "could not add expense")
		result := getAllExpenses(t, db).FindByID(expense.ID)
		if assert.NotNil(t, result, "expense not added") {
			assert.Equal(t, expense, *result, "added expense is different")
		}
	})
}
```

---

### Testing APIs

![](images/d9e42a4a63c2fa4815b4d65a98b198fd.png)

```go
func TestAPI(t *testing.T) {
	server := startServer(t, ctx)

t.Run("summarize expenses", func(t *testing.T) {
		response, responseBody := call(t, server, http.MethodGet, "/expenses/summarize", "")

assert.Equal(t, http.StatusOK, response.StatusCode, "status code")
		expected := getExpectedResponse(t)
		// Or use https://github.com/kinbiko/jsonassert
		assert.JSONEq(t, expected, responseBody, "response body")
	})
	t.Run("add expense fails", func(t *testing.T) {
		response, responseBody := call(t, server, http.MethodPost, "/expenses/add", getRequest(t))

assert.Equal(t, http.StatusBadRequest, response.StatusCode, "status code")
		expected := getExpectedResponse(t)
		assert.JSONEq(t, expected, responseBody, "response body")
        
		result := getAllExpenses(t, db).FindByID("...")
		assert.Nil(t, result, "expense added")
	})
}

func startServer(t *testing.T, ctx context.Context) *httptest.Server {
	dbContainer := test_repos.StartDB(t, ctx)
	err := os.Setenv("DB_URL", dbContainer.DSN)
	// ...
	setup, err := api.NewSetup()
	// ...
	return httptest.NewServer(setup.APIMux)
}
```

---

### Go testing goodies

```go
func startServer(t *testing.T, ctx context.Context) *httptest.Server {
	// ...
}

func call(
	t *testing.T,
	srv *httptest.Server,
	method, path, body string,
) (*http.Response, string) {
	// ...
}

func getRequest(t *testing.T) string {
	// ...
}

func getExpectedResponse(t *testing.T) string {
	// ...
}
```

How do we get here?

### `t.Name()`

```go
func getRequest(t *testing.T, requestPath string) (string, error) {
	t.Helper()

file, err := os.ReadFile(requestPath)
	if err != nil {
		return "", err
	}

return string(file), nil
}
```

```go
func getRequest(t *testing.T) (string, error) {
	t.Helper()

path := fmt.Sprintf("./testdata/%s/request.json", t.Name())
	file, err := os.ReadFile(path)
	if err != nil {
		return "", err
	}

return string(file), nil
}
```

![](images/e64b7561cc407ecca7a9bb0e4b965d0b.png)

### `t.FailNow()`

```go
func getRequest(t *testing.T) (string, error) {
	t.Helper()

path := fmt.Sprintf("./testdata/%s/request.json", t.Name())
	file, err := os.ReadFile(path)
	if err != nil {
		return "", err
	}

return string(file), nil
}
```

```go
func getRequest(t *testing.T) string {
	t.Helper()

path := fmt.Sprintf("./testdata/%s/request.json", t.Name())
	file, err := os.ReadFile(path)
	require.NoError(t, err, "read file")

return string(file)
}
```

### `t.Cleanup()`

```go
func startServer(t *testing.T, ctx context.Context) (*httptest.Server, func()) {
	t.Helper()

// ...

server := httptest.NewServer(setup.APIMux)

return server, server.Close
}
```

```go
func startServer(t *testing.T, ctx context.Context) *httptest.Server {
	t.Helper()

// ...

server := httptest.NewServer(setup.APIMux)
	t.Cleanup(func() {
		server.Close()
	})

return server
}
```

### `t.Failed()`

```go
t.Cleanup(func() {
	// When test fail, printing logs is usually helpful :)
	if t.Failed() {
		reader, _ := getServerContainer(t, ctx, compose).Logs(ctx)
		bytes, _ := io.ReadAll(reader)
		fmt.Println(`\nLogs from "server" container:\n`, string(bytes))
	}
	assert.NoError(t, compose.Down(ctx, tc.RemoveOrphans(true), tc.RemoveImagesLocal))
})
```

### Crispy clear tests, where helpers just disappear

```go
func TestAPI(t *testing.T) {
	server := startServer(t, ctx)
    
	t.Run("add expense fails", func(t *testing.T) {
		response, responseBody := call(
			t, server, http.MethodPost, "/expenses/add", getRequest(t),
		)

assert.Equal(t, http.StatusBadRequest, response.StatusCode, "status code")
		expected := getExpectedResponse(t)
		assert.JSONEq(t, expected, responseBody, "response body")
	})
}
```

Note: If you have sent this to code review, they wouldn't notice. A lot of good code is simple, "obvious". It doesn't mean it's easy to get there.

---

### Testing entire app/service

![](images/748d64661b10ae6936501ad368299150.png)

The original idea came from service like this. It was super annoying to test.

Let's see how our example app works

![](images/624e646a3fdafb349851c3bbe43a404e.png)

And this is a diagram of how test works

![](images/c54f08a4cd2985789699cdcaa85dbf0c.png)

```yaml
version: "3.9"

services:
  server:
    image: "expense_tracker/server:latest"
    build:
      context: ./..
      dockerfile: app_to_test/server/Dockerfile
    environment:
      DB_URL: "postgres://postgres:secret123@db:5432/expense_tracker?sslmode=disable"
      BANK_API_URL: "${BANK_API_URL}"
    depends_on:
      db:
        condition: service_healthy
    # Thanks to this container can call our mock HTTP server on host machine.
    extra_hosts:
      - "host.docker.internal:host-gateway"
    ports:
      # Once again, we use a random port to avoid conflicts.
      - "8000"

db:
    image: "postgres:15.2-alpine"
    environment:
      POSTGRES_DB: expense_tracker
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: secret123
    healthcheck:
      test: [ "CMD-SHELL", "pg_isready -d $${POSTGRES_DB} -U $${POSTGRES_USER}" ]
      interval: 3s
      timeout: 60s
      retries: 10
      start_period: 5s

migrate:
    image: "expense_tracker/migrate:latest"
    build:
      context: ./../app_to_test/db
    environment:
      DB_URL: "postgres://postgres:secret123@db:5432/expense_tracker?sslmode=disable"
    depends_on:
      db:
        condition: service_healthy
```

```go
const expenseToSyncID = "677df0c4-d829-42eb-a0c9-29d5b0a2bbe4"

func TestE2E(t *testing.T) {
	ctx := context.Background()

bankAPIAddress := mockBankAPI(t)
	address := startApp(t, ctx, bankAPIAddress)

t.Run("app is starting properly", func(t *testing.T) {
		response, err := http.Get(fmt.Sprintf("%s/expenses/all", address))

require.NoError(t, err)
		assert.Equal(t, http.StatusOK, response.StatusCode, "status code")
	})
	t.Run("sync expenses", func(t *testing.T) {
		response, err := http.Get(fmt.Sprintf("%s/expenses/sync", address))

require.NoError(t, err)
		require.Equal(t, http.StatusOK, response.StatusCode, "status code")

response, err = http.Get(fmt.Sprintf("%s/expenses/all", address))
		require.NoError(t, err)
		responseBody, err := io.ReadAll(response.Body)
		require.NoError(t, err)
		assert.Contains(t, string(responseBody), expenseToSyncID)
	})
}
```

```go
func mockBankAPI(t *testing.T) (address string) {
	t.Helper()

mux := http.NewServeMux()
	mux.Handle("/get-transactions", http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		w.Write([]byte(fmt.Sprintf(`
			[
			  {
				"id": "%s",
				"amount": 500.00,
				"category": "food",
				"created_at": "2020-01-01T00:00:00Z"
			  }
			]`,
			expenseToSyncID)))
	}))

server := httptest.NewUnstartedServer(mux)
	listener, err := net.Listen("tcp4", "0.0.0.0:0")
	require.NoError(t, err, "could not start listener")
	addr, err := net.ResolveTCPAddr(listener.Addr().Network(), listener.Addr().String())
	require.NoError(t, err, "could not resolve tcp addr")
	server.Listener = listener
	server.Start()
	t.Cleanup(func() {
		server.Close()
	})

return fmt.Sprintf("http://host.docker.internal:%d", addr.Port)
}
```

```go
func startApp(t *testing.T, ctx context.Context, bankAPIAddress string) (address string) {
	t.Helper()

compose, err := tc.NewDockerComposeWith(
		tc.WithStackFiles("./docker-compose-for-e2e.yaml"),
		// Giving unique name to each docker compose stack allows us to run tests in parallel.
		tc.StackIdentifier(uuid.New().String()),
	)
	require.NoError(t, err, "docker compose setup")

t.Cleanup(func() {
		// When test fail, printing logs is usually helpful :)
		if t.Failed() {
			reader, _ := getServerContainer(t, ctx, compose).Logs(ctx)
			bytes, _ := io.ReadAll(reader)
			fmt.Println(`\nLogs from "server" container:\n`, string(bytes))
		}
		assert.NoError(t, compose.Down(ctx, tc.RemoveOrphans(true), tc.RemoveImagesLocal))
	})

err = compose.
		WithEnv(map[string]string{
			"BANK_API_URL": bankAPIAddress,
		}).
		WaitForService("server", wait.ForLog("running...")).
		Up(ctx)
	require.NoError(t, err, "docker compose up")

// Port is randomly assigned by docker. We need to get it.
	apiPort, err := getServerContainer(t, ctx, compose).MappedPort(ctx, "8000")
	require.NoError(t, err, "docker compose server port")

return fmt.Sprintf("http://localhost:%s", apiPort.Port())
}
```

---

# Thanks!

**Question for you:** Should you start containers within `TestMain`?

**Any questions for me?**

![](images/2df0e65d2dae5691a7f6a31ebdaf5f79.png)

![](images/046a5ac1c00d1c39a16b375aa1e5e192.svg)

[https://github.com/antosdaniel/go-presentation-beyond-unit-tests](https://github.com/antosdaniel/go-presentation-beyond-unit-tests)