Go Web Application Structure - Part 4 - Database Migrations & Business Logic

Go Web Application Structure - Part 4 - Database Migrations & Business Logic
  • Part 1: Web Application Structure
  • Part 2: Routing/Serving
  • Part 3: The Database
  • Part 4: Database Migrations & Business Logic (You're looking at it!)

Introduction

In the previous part, we talked about how to interface with our PostgresSQL database. Now we will go over how to run database migrations as well as properly add business logic to our application.

Database Migrations

Frameworks like Django and Rails are able to generate migration files based on the data models you define. Gorm doesn't have this feature, however, it does have some interfaces to make our lives easier when creating migrations.

The AutoMigrate

Gorm has a method where we can "auto migrate" the database, however, there are some major caveats to this:

AutoMigrate will ONLY create tables, missing columns and missing indexes, and WON’T change existing column’s type or delete unused columns to protect your data.

This is ok as we will implement our own migrations.

Custom Migrations

The code for our migrate command is a little too long to add inline for this post, so view the full code on Github. The basic idea is this:

./todos migrate --dry-run
./todos migrate

To do this we need to create a new migrations folder with mkdir -p $(go env GOPATH)/src/github.com/<username>/todos/migrations. And we will also create a new file called migrate.go in the cmd directory.

In migrations/migration.go we have a simple struct for storing the migration data:

package migrations

import (
	"github.com/jinzhu/gorm"
)

type Migration struct {
	Number uint `gorm:"primary_key"`
	Name   string

	Forwards func(db *gorm.DB) error `gorm:"-"`
}

var Migrations []*Migration

We will be able to see what migrations would be run using the --dry-run flag and then just run all unapplied migrations. To define new migrations, we can do this:

// migrations/0001_add_user.go
package migrations

import (
	"github.com/jinzhu/gorm"
	"github.com/pkg/errors"
)

var addUserMigration_0001 = &Migration{
	Number: 1,
	Name:   "Add user",
	Forwards: func(db *gorm.DB) error {
		const addUserSQL = `
			CREATE TABLE users(
 				id serial PRIMARY KEY,
 				email text UNIQUE NOT NULL,
				hashed_password bytea NOT NULL,
 				created_at TIMESTAMP NOT NULL,
 				updated_at TIMESTAMP NOT NULL,
 				deleted_at TIMESTAMP
			);
		`

		err := db.Exec(addUserSQL).Error
		return errors.Wrap(err, "unable to create users table")
	},
}

func init() {
	Migrations = append(Migrations, addUserMigration_0001)
}

Then our migrate command will look at the migrations.Migrations array to apply the proper migrations. For demonstration purposes, we only implemented forward migrations. Occasionally when working locally you want to rollback already applied migrations; this would be trivial to implement with a new Backwards field on the Migration struct.

We now have database migrations! When modifying our schema we would create a new file in the migrations directory with an incremented number.

Business Logic

We talked about what "business logic" was back in Part 1 of this series, but here's a quick recap:

Business logic or domain logic is that part of the program which encodes the real-world business rules that determine how data can be created, displayed, stored, and changed. It prescribes how business objects interact with one another, and enforces the routes and the methods by which business objects are accessed and updated.

Authentication

For our demonstration purposes, we will implement basic http authentication. For a "real world" api, with many requests, this will be very expensive. It is worth reading more about things like JSON Web Tokens.

First, let's update our Context struct to handle a user.

// ... snip ...

type Context struct {
	Logger        logrus.FieldLogger
	RemoteAddress string
	Database      *db.Database
	User          *model.User // NEW
}

// ... snip ...

func (ctx *Context) WithUser(user *model.User) *Context {
	ret := *ctx
	ret.User = user
	return &ret
}

We can add a few lines to our API.handler method to check for authentication:

// ... snip ...

func (a *API) handler(f func(*app.Context, http.ResponseWriter, *http.Request) error) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		// ... snip ...

		ctx := a.App.NewContext().WithRemoteAddress(a.IPAddressForRequest(r))
		ctx = ctx.WithLogger(ctx.Logger.WithField("request_id", base64.RawURLEncoding.EncodeToString(model.NewId())))

		if username, password, ok := r.BasicAuth(); ok {
			user, err := a.App.GetUserByEmail(username)

			if user == nil || err != nil {
				if err != nil {
					ctx.Logger.WithError(err).Error("unable to get user")
				}
				http.Error(w, "invalid credentials", http.StatusForbidden)
				return
			}

			if ok := user.CheckPassword(password); !ok {
				http.Error(w, "invalid credentials", http.StatusForbidden)
				return
			}

			ctx = ctx.WithUser(user)
		}

        // ... snip ...
	})
}

// ... snip ...

New Context Methods

Users

For users, we will create a couple of new methods. The first is we want to create a user. When a user is created we want to validate they have a valid email and they supplied a password.

// ... snip ...

func (ctx *Context) CreateUser(user *model.User, password string) error {
	if err := ctx.validateUser(user, password); err != nil {
		return err
	}

	if err := user.SetPassword(password); err != nil {
		return errors.Wrap(err, "unable to set user password")
	}

	return ctx.Database.CreateUser(user)
}

func (ctx *Context) validateUser(user *model.User, password string) *ValidationError {
	// naive email validation
	if !strings.Contains(user.Email, "@") {
		return &ValidationError{"invalid email"}
	}

	if password == "" {
		return &ValidationError{"password is required"}
	}

	return nil
}

In our validateUser method, we return a special error that we created called ValidationError. In our api method handler, we can now check for this specifically and return the json representation along with a 400 status code instead of a 500.

// ... snip ...

func (a *API) handler(f func(*app.Context, http.ResponseWriter, *http.Request) error) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		// ... snip ...

		w.Header().Set("Content-Type", "application/json")

		if err := f(ctx, w, r); err != nil {
			if verr, ok := err.(*app.ValidationError); ok {
				data, err := json.Marshal(verr)
				if err == nil {
					w.WriteHeader(http.StatusBadRequest)
					_, err = w.Write(data)
				}

				if err != nil {
					ctx.Logger.Error(err)
					http.Error(w, "internal server error", http.StatusInternalServerError)
				}
			} else if uerr, ok := err.(*app.UserError); ok {
				data, err := json.Marshal(uerr)
				if err == nil {
					w.WriteHeader(uerr.StatusCode)
					_, err = w.Write(data)
				}

				if err != nil {
					ctx.Logger.Error(err)
					http.Error(w, "internal server error", http.StatusInternalServerError)
				}
			} else {
				ctx.Logger.Error(err)
				http.Error(w, "internal server error", http.StatusInternalServerError)
			}
		}
	})
}

// ... snip ...

Todos

We will need to create a few more methods now to handle interacting with Todos. In app/todo.go we will create the following:

package app

import "github.com/theaaf/todos/model"

func (ctx *Context) GetTodoById(id uint) (*model.Todo, error) {
	if ctx.User == nil {
		return nil, ctx.AuthorizationError()
	}

	todo, err := ctx.Database.GetTodoById(id)
	if err != nil {
		return nil, err
	}

	if todo.UserID != ctx.User.ID {
		return nil, ctx.AuthorizationError()
	}

	return todo, nil
}

func (ctx *Context) getTodosByUserId(userId uint) ([]*model.Todo, error) {
	return ctx.Database.GetTodosByUserId(userId)
}

func (ctx *Context) GetUserTodos() ([]*model.Todo, error) {
	if ctx.User == nil {
		return nil, ctx.AuthorizationError()
	}

	return ctx.getTodosByUserId(ctx.User.ID)
}

func (ctx *Context) CreateTodo(todo *model.Todo) error {
	if ctx.User == nil {
		return ctx.AuthorizationError()
	}

	todo.UserID = ctx.User.ID

	if err := ctx.validateTodo(todo); err != nil {
		return err
	}

	return ctx.Database.CreateTodo(todo)
}

const maxTodoNameLength = 100

func (ctx *Context) validateTodo(todo *model.Todo) *ValidationError {
	if len(todo.Name) > maxTodoNameLength {
		return &ValidationError{"name is too long"}
	}

	return nil
}

func (ctx *Context) UpdateTodo(todo *model.Todo) error {
	if ctx.User == nil {
		return ctx.AuthorizationError()
	}

	if todo.UserID != ctx.User.ID {
		return ctx.AuthorizationError()
	}

	if err := ctx.validateTodo(todo); err != nil {
		return nil
	}

	return ctx.Database.UpdateTodo(todo)
}

func (ctx *Context) DeleteTodoById(id uint) error {
	if ctx.User == nil {
		return ctx.AuthorizationError()
	}

	todo, err := ctx.GetTodoById(id)
	if err != nil {
		return err
	}

	if todo.UserID != ctx.User.ID {
		return ctx.AuthorizationError()
	}

	return ctx.Database.DeleteTodoById(id)
}

This is a little bit more interesting than with the user. In these methods, we have checking to make sure that the current user associated with the request can only operate on todos that they created. And in CreateTodo we even make doubly sure that the todo being created is associated with the current user.

Why have a custom context struct?

This was asked in Part 2 of the series and it's a valid question. An http.Request has its own context that you can access via r.Context(). This is great! However, if we wanted to store 4 different values like we are in our context we would need to use type assertions with each call to ctx.Value("database"). There have been some benchmarks run that discuss the performance of type assertions. It appears that after Go 1.4, the performance improved dramatically. I think that having our own context struct makes interacting with a request a little bit more natural. Instead of having methods defined like this:

func GetTodoById(ctx context.Context, id uint) (*model.Todo, error) {
	user, ok := ctx.Value("user").(*model.User)
	if !ok {
		return nil, &UserError{Message: "unauthorized", StatusCode: http.StatusForbidden}
	}

	db, _ := ctx.Value("database").(*db.Database)

	todo, err := db.GetTodoById(id)
	if err != nil {
		return nil, err
	}

	if todo.UserID != user.ID {
		logger := ctx.Value("logger").(logrus.FieldLogger)
		logger.Warn("user trying to access random todo")
		return nil, &UserError{Message: "unauthorized", StatusCode: http.StatusForbidden}
	}

	return todo, nil
}

With our own Context struct we don't need to have all of these ugly type assertions all over our code and we can feel more confident of these values actually existing in the context. If we were using a library like gqlgen where the request context is passed to each resolver, we could set our custom context value in the request context and then everything would work like before.

To answer the question, it's a matter of preference. I prefer to have a formalized Context struct, but if you would rather use type assertions then that's ok too. There is also some debate in the Go community around the context package.

API Routes

Now that all of the context methods are defined, we need to expose them to the user. We will do this through a REST API, however, there are other alternatives like GraphQL or gRPC.

User Routes

Similar to our context methods, we will group them by their use. For users, we will create an api/user.go file.

package api

import (
	"encoding/json"
	"io/ioutil"
	"net/http"

	"github.com/theaaf/todos/app"
	"github.com/theaaf/todos/model"
)

type UserInput struct {
	Email    string `json:"email"`
	Password string `json:"password"`
}

type UserResponse struct {
	Id uint `json:"id"`
}

func (a *API) CreateUser(ctx *app.Context, w http.ResponseWriter, r *http.Request) error {
	var input UserInput

	defer r.Body.Close()
	body, err := ioutil.ReadAll(r.Body)
	if err != nil {
		return err
	}

	if err := json.Unmarshal(body, &input); err != nil {
		return err
	}

	user := &model.User{Email: input.Email}

	if err := ctx.CreateUser(user, input.Password); err != nil {
		return err
	}

	data, err := json.Marshal(&UserResponse{Id: user.ID})
	if err != nil {
		return err
	}

	_, err = w.Write(data)
	return err
}

Here we parse the incoming JSON, create the user, and finally return a json representation of the user. The next step is to update our API.Init method.

// ... snip ...

func (a *API) Init(r *mux.Router) {
	// user methods
	r.Handle("/users/", a.handler(a.CreateUser)).Methods("POST")
}

// ... snip ...

We can test this with cURL or httpie.
Screen-Shot-2019-03-21-at-2.53.22-PM

Todo Routes

Starting with listing todos, we can create a new file api/todo.go.

// ... snip ...

func (a *API) GetTodos(ctx *app.Context, w http.ResponseWriter, r *http.Request) error {
	todos, err := ctx.GetUserTodos()
	if err != nil {
		return err
	}

	data, err := json.Marshal(todos)
	if err != nil {
		return err
	}

	_, err = w.Write(data)
	return err
}

Screen-Shot-2019-03-21-at-3.04.35-PM

We haven't created any Todos yet, so we just get an empty list back. Let's add that next:

// ... snip ...
type CreateTodoInput struct {
	Name string `json:"name"`
	Done bool   `json:"done"`
}

type CreateTodoResponse struct {
	Id uint `json:"id"`
}

func (a *API) CreateTodo(ctx *app.Context, w http.ResponseWriter, r *http.Request) error {
	var input CreateTodoInput

	defer r.Body.Close()
	body, err := ioutil.ReadAll(r.Body)
	if err != nil {
		return err
	}

	if err := json.Unmarshal(body, &input); err != nil {
		return err
	}

	todo := &model.Todo{Name: input.Name, Done: input.Done}

	if err := ctx.CreateTodo(todo); err != nil {
		return err
	}

	data, err := json.Marshal(&CreateTodoResponse{Id: todo.ID})
	if err != nil {
		return err
	}

	_, err = w.Write(data)
	return err
}

Now we can add a few Todos and fetch them again.
Screen-Shot-2019-03-21-at-3.12.17-PM
Screen-Shot-2019-03-21-at-3.13.04-PM

Next let's allow updating of Todos:

// ... snip ...
type UpdateTodoInput struct {
	Name *string `json:"name"`
	Done *bool   `json:"done"`
}

func (a *API) UpdateTodoById(ctx *app.Context, w http.ResponseWriter, r *http.Request) error {
	id := getIdFromRequest(r)

	var input UpdateTodoInput

	defer r.Body.Close()
	body, err := ioutil.ReadAll(r.Body)
	if err != nil {
		return err
	}

	if err := json.Unmarshal(body, &input); err != nil {
		return err
	}

	existingTodo, err := ctx.GetTodoById(id)
	if err != nil || existingTodo == nil {
		return err
	}

	if input.Name != nil {
		existingTodo.Name = *input.Name
	}
	if input.Done != nil {
		existingTodo.Done = *input.Done
	}

	err = ctx.UpdateTodo(existingTodo)
	if err != nil {
		return err
	}

	data, err := json.Marshal(existingTodo)
	if err != nil {
		return err
	}

	_, err = w.Write(data)
	return err
}

And finally delete a Todo:

func (a *API) DeleteTodoById(ctx *app.Context, w http.ResponseWriter, r *http.Request) error {
	id := getIdFromRequest(r)
	err := ctx.DeleteTodoById(id)

	if err != nil {
		return err
	}

	return &app.UserError{StatusCode: http.StatusOK, Message: "removed"}
}

Let's test it out!

Screen-Shot-2019-03-21-at-3.29.10-PM

Looks like it's working as expected. Of course, we can't expect users to interact with the API through the command line. The next step would be to create the user interface!

Recap

In this last part, we covered a lot. However, there is still room for improvement. As you may have noticed, there are some duplicated patterns in our api handlers for marshaling and unmarshaling json. Of course, using REST isn't the only option when designing an api. At AAF we use GraphQL and have had a very good experience with it. In order to make the right decision for your API, there are many things to consider, but we'll leave that for another post.


All the code for this part can be viewed here. Note that if the code on Github is updated, these blog posts may become out of date.


Photo by rawpixel.com from Pexels

Go Web Application Structure - Part 4 - Database Migrations & Business Logic
Share this