learning-go

Learn Go by building a REST API and a Command Line Interface (CLI)

Add authentication to the API

The goal of this section is to change the API to have an authentication based on JSON Web Tokens (JWT) generated by our OAuth2 server (Amazon Cognito).

To achieve that we need:

  1. Create a Gin middleware to intercept all requests and check the JWT
  2. Verify the JWT as described in the AWS Documentation

This is the feature more complex to implement and to explain. If you fell it is hard to understand please open an issue with your questions/difficulties.

Base structure of the middleware

The authentication middleware will:

  1. Get the JWT from the Authentication HTTP header
  2. Validates the JWT
  3. If the token is valid, put the client identifier in the Gin context, in case any other feature requires it
  4. Abort the request and return an HTTP 401 status in case of any issue

The basic structure of the middleware is:

func Authenticator(ac *AuthenticatorConfig) gin.HandlerFunc {
    return func(c *gin.Context) {
        // Ignore the root as it is used for the liveness probes
        if c.Request.URL.Path == "/" {
            return
        }

        // Gets the JWT from the Authentication header
        authHeader := c.GetHeader("Authentication")
        if authHeader == "" {
            log.Debug().Msg("JWT not found")
            c.AbortWithStatusJSON(
                http.StatusUnauthorized,
                apierror.New("Not authorized"))
            return
        }

        // Validates the JWT
        token, err := validateToken(ac, authHeader)
        if err != nil {
            log.Debug().Err(err).Msg("JWT not valid")
            c.AbortWithStatusJSON(
                http.StatusUnauthorized,
                apierror.New("Not authorized"))
            return
        }

        // Put the client identifier in the Gin context
        ci, _ := token.Get(ClientIdKey)
        c.Set(ClientIdKey, ci)
    }
}

The code at this point is fairly simple and straightforward. I just would like to highlight the AuthenticatorConfig structure. We needed it because the token validation requires additional data, set in the environment variables used to run the API:

  1. The JSON Web Key Set (JWKS), a set of keys containing the public keys used to verify any JSON Web Token (JWT) issued by the authorization server and signed using the RS256 signing algorithm.
  2. The identifier of the Cognito user pool, used to validate the issuer (iss) claim

Make the middleware verify the JWT

The contents of the internal/middleware/authenticator.go file is:

package middleware

import (
    "fmt"
    "net/http"

    "github.com/gin-gonic/gin"
    "github.com/lestrrat-go/jwx/jwk"
    "github.com/lestrrat-go/jwx/jwt"
    "github.com/renato0307/learning-go-api/internal/apierror"
    "github.com/rs/zerolog/log"
)

type AuthenticatorConfig struct {
    KeySetJSON []byte
    Issuer     string
}

const (
    TokenUseKey = "token_use"
    ClientIdKey = "client_id"
)

func Authenticator(ac *AuthenticatorConfig) gin.HandlerFunc {
    return func(c *gin.Context) {
        // Ignore the root as it is used for the liveness probes
        if c.Request.URL.Path == "/" {
            return
        }

        // Gets the JWT from the Authentication header
        authHeader := c.GetHeader("Authentication")
        if authHeader == "" {
            log.Debug().Msg("JWT not found")
            c.AbortWithStatusJSON(
                http.StatusUnauthorized,
                apierror.New("Not authorized"))
            return
        }

        // Validates the JWT
        token, err := validateToken(ac, authHeader)
        if err != nil {
            log.Debug().Err(err).Msg("JWT not valid")
            c.AbortWithStatusJSON(
                http.StatusUnauthorized,
                apierror.New("Not authorized"))
            return
        }

        // Put the client identifier in the Gin context
        ci, _ := token.Get(ClientIdKey)
        c.Set(ClientIdKey, ci)
    }
}

func validateToken(ac *AuthenticatorConfig, tokenString string) (jwt.Token, error) {
    keySet, err := jwk.Parse(ac.KeySetJSON)
    if err != nil {
        return nil, fmt.Errorf("failed to parse keyset: %s", err)
    }

    // Step 1: Confirm the structure of the JWT
    // Step 2: Validate the JWT signature
    token, err := jwt.Parse(
        []byte(tokenString),
        jwt.WithKeySet(keySet),
    )
    if err != nil {
        log.Debug().Err(err).Msg("error parsing the token")
        return nil, fmt.Errorf("invalid token: %s", err)
    }

    // Step 3: Verify the claims
    clientId, _ := token.Get(ClientIdKey)
    err = jwt.Validate(token,
        jwt.WithClaimValue(TokenUseKey, "access"),
        jwt.WithClaimValue(jwt.IssuerKey, ac.Issuer),
        jwt.WithRequiredClaim(ClientIdKey),
        jwt.WithRequiredClaim(jwt.SubjectKey),
        jwt.WithClaimValue(jwt.SubjectKey, clientId),
    )
    if err != nil {
        log.Debug().Err(err).Msg("error validating the token")
        return nil, fmt.Errorf("invalid token: %s", err)
    }

    return token, nil
}

Let me highlight a couple of things:

  1. First we load the JWKS (JSON Web Key Set), which contains the key to used to compare the signature of the issuer to the signature in the token.
  2. The jwt.Parse checks the JWT structure and uses the JWKS to verify the signature.
  3. The jwt.Validate is used to validate the claim as recommended in the AWS Documentation

Unit testing for the middleware

The unit tests for the middleware are a bit complex because we need to generate keys and JSON Web Key Set. Additionally we are going to use a new technique to cover all the needed cases, keeping the code DRYier.

Let's highlight the most important aspects:

  1. The TestAuthenticatorNoAuthHeader function is pretty straightforward and it doesn't deserve much comments.
  2. The TestAuthenticatorRootPathSkipsAuth function checks if the root path has no authentication because we need it for the liveness probes.
  3. The TestAuthenticatorWithJWT function creates an array of test cases related with the validation of the JWT and the claims. Each element of the array contains the JWT to be checked, the expected status code for the JWT, the purpose of the test and the contents of the body.
  4. The test cases are executed using a for loop, by initializing Gin, making a simple request and checking the results.
  5. The rest of the internal/middleware/authenticator_test.go file handles the creation of the keys, the token and the sign process so the test cases can be executed.
package middleware

import (
    "crypto/rand"
    "crypto/rsa"
    "encoding/json"
    "fmt"
    "net/http"
    "testing"
    "time"

    "github.com/gin-gonic/gin"
    "github.com/lestrrat-go/jwx/jwa"
    "github.com/lestrrat-go/jwx/jwk"
    "github.com/lestrrat-go/jwx/jwt"
    "github.com/renato0307/learning-go-api/internal/apierror"
    "github.com/renato0307/learning-go-api/internal/apitesting"
    "github.com/stretchr/testify/assert"
)

const userPool string = "https://cognito-idp.eu-west-1.amazonaws.com/eu-west-1_xxxxxxxxxx"

func TestAuthenticatorNoAuthHeader(t *testing.T) {

    // arrange - init gin to use the structured logger middleware
    r := gin.New()
    r.Use(Authenticator(nil))
    r.Use(gin.Recovery())

    // arrange - set the routes
    r.GET("/example", func(c *gin.Context) {})

    // act
    w := apitesting.PerformRequest(r, "GET", "/example?a=100")

    // assert
    assert.Equal(t, http.StatusUnauthorized, w.Code)
    apierror.AssertIsValid(t, w.Body.Bytes())
}

func TestAuthenticatorRootPathSkipsAuth(t *testing.T) {

    // arrange - init gin to use the structured logger middleware
    r := gin.New()
    r.Use(Authenticator(nil))
    r.Use(gin.Recovery())

    // arrange - set the routes
    r.GET("/", func(c *gin.Context) {})

    // act
    w := apitesting.PerformRequest(r, "GET", "/")

    // assert
    assert.Equal(t, http.StatusOK, w.Code)
}

func TestAuthenticatorWithJWT(t *testing.T) {
    // arrange - generate key, keyset and JWT
    key := generateKey(t)
    jwks := generateKeySetInJSON(&key, t)

    // arrange - define the several test cases
    testCases := []struct {
        JWT          string
        StatusCode   int
        Purpose      string
        BodyContains string
    }{
        {
            JWT: "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9." +
                "eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ." +
                "SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c",
            StatusCode:   http.StatusUnauthorized,
            Purpose:      "token with signed with another key",
            BodyContains: "Not authorized",
        },
        {
            JWT:          newJWT(key, true, false, false, t),
            StatusCode:   http.StatusUnauthorized,
            Purpose:      "invalid subject claim",
            BodyContains: "Not authorized",
        },
        {
            JWT:          newJWT(key, false, true, false, t),
            StatusCode:   http.StatusUnauthorized,
            Purpose:      "invalid expiration claim",
            BodyContains: "Not authorized",
        },
        {
            JWT:          newJWT(key, false, false, true, t),
            StatusCode:   http.StatusUnauthorized,
            Purpose:      "invalid token_use claim",
            BodyContains: "Not authorized",
        },
        {
            JWT:          newValidJWT(key, t),
            StatusCode:   http.StatusOK,
            Purpose:      "valid JWT",
            BodyContains: "",
        },
    }

    for _, tc := range testCases {

        // arrange - init gin to use the authenticator middleware
        r := gin.New()
        authConfig := AuthenticatorConfig{
            KeySetJSON: jwks,
            Issuer:     userPool,
        }
        r.Use(Authenticator(&authConfig))
        r.Use(gin.Recovery())

        // arrange - set the routes
        r.GET("/example", func(c *gin.Context) {
            cid, _ := c.Get(ClientIdKey)
            c.JSON(http.StatusOK, cid)
        })

        // arrange - headers
        header := http.Header{}
        header.Add("Authentication", tc.JWT)

        // act
        w := apitesting.PerformRequestWithHeader(r, "GET", "/example?a=100", header)

        // assert
        assert.Equal(t,
            tc.StatusCode,
            w.Code,
            fmt.Sprintf("failed %s", tc.Purpose))

        b := w.Body.String()
        assert.Contains(t, b, tc.BodyContains)
    }
}

func newValidJWT(key jwk.Key, t *testing.T) string {
    return newJWT(key, false, false, false, t)
}

func newJWT(key jwk.Key, noSub, noExp, noTokenUse bool, t *testing.T) string {
    token := jwt.New()
    if !noSub {
        token.Set("sub", "client_id_1234567890")
    }
    if !noTokenUse {
        token.Set("token_use", "access")
    }
    token.Set("scope", "https://learninggolang.com/all")
    token.Set("auth_time", 1641417382)
    token.Set("iss", userPool)
    if !noExp {
        token.Set("exp", time.Now().Unix()+1000)
    } else {
        token.Set("exp", 1)
    }
    token.Set("iat", 1641417382)
    token.Set("version", 2)
    token.Set("jti", "a6dd28cc-500e-4b49-a510-efda5195d2f4")
    token.Set("client_id", "client_id_1234567890")

    signed, err := signJWT(token, key)
    if err != nil {
        t.Fatal(err)
    }

    return string(signed)
}

func signJWT(token jwt.Token, key jwk.Key) ([]byte, error) {
    return jwt.Sign(token, jwa.RS256, key)
}

func generateKey(t *testing.T) jwk.Key {
    raw, err := rsa.GenerateKey(rand.Reader, 2048)
    if err != nil {
        t.Fatalf("failed to generate new RSA private key: %s\n", err)
    }

    key, err := jwk.New(raw)
    if err != nil {
        t.Fatalf("failed to create symmetric key: %s\n", err)
    }
    if _, ok := key.(jwk.RSAPrivateKey); !ok {
        t.Fatalf("expected jwk.SymmetricKey, got %T\n", key)
    }

    key.Set(jwk.KeyIDKey, "mykey")

    return key
}

func generateKeySetInJSON(key *jwk.Key, t *testing.T) []byte {
    set := jwk.NewSet()
    pubKey, _ := (*key).(jwk.RSAPrivateKey).PublicKey()
    pubKey.Set(jwk.AlgorithmKey, "RS256")
    set.Add(pubKey)

    buf, err := json.MarshalIndent(set, "", "  ")
    if err != nil {
        t.Fatalf("failed to marshal key into JSON: %s\n", err)
    }

    return buf
}

The test requires a new package created to store helper functions for testing.

We are going to put the code in the internal/apitesting/helper.go file.

mkdir -p internal/apitesting
touch internal/apitesting/helper.go

The file contents are:

package apitesting

import (
	"net/http"
	"net/http/httptest"
)

// PerformRequest executes an http request for testing
func PerformRequest(r http.Handler, method, path string) *httptest.ResponseRecorder {
	return PerformRequestWithHeader(r, method, path, nil)
}

// PerformRequestWithHeader executes an http request for testing with header
// support
func PerformRequestWithHeader(
	r http.Handler,
	method, path string,
	header http.Header) *httptest.ResponseRecorder {

	req := httptest.NewRequest(method, path, nil)
	req.Header = header
	w := httptest.NewRecorder()
	r.ServeHTTP(w, req)
	return w
}

🕵️‍♀️ GO-EXTRA: Defining structures inline

From the last code block we are using structures in different way: we declare and initialize them inline.

testCases := []struct {
    JWT          string
    StatusCode   int
    Purpose      string
    BodyContains string
}{
    // ...
    {
        JWT:          newJWT(key, true, false, false, t),
        StatusCode:   http.StatusUnauthorized,
        Purpose:      "invalid subject claim",
        BodyContains: "Not authorized",
    },
    {
        JWT:          newJWT(key, false, true, false, t),
        StatusCode:   http.StatusUnauthorized,
        Purpose:      "invalid expiration claim",
        BodyContains: "Not authorized",
    },
    // ...
}

Make Gin use the middleware

We need the following changes in the main.go file:

// ..
func main() {
    // Initialize Gin
    gin.SetMode(gin.ReleaseMode)
    r := gin.New()
    r.Use(middleware.DefaultStructuredLogger())
    r.Use(middleware.Authenticator(newAuthenticatorConfig())) // new
    r.Use(gin.Recovery())
    //...

The newAuthenticatorConfig function does all the work:

// newAuthenticatorConfig gathers all authentication related information to set
// up the Authenticator middleware configuration.
//
// It requires the definition two environment variables:
//
// AUTH_JWKS_LOCATION: https://cognito-idp.$AWS_REGION.amazonaws.com/$POOL_ID/.well-known/jwks.json
//
// AUTH_TOKEN_ISS: https://cognito-idp.$AWS_REGION.amazonaws.com/$POOL_ID
func newAuthenticatorConfig() *middleware.AuthenticatorConfig {

    // Gets the JSON Web Key Set download URL
    jwksLocation := getRequiredEnv(AUTH_JWKS_LOCATION)
    r, err := http.Get(jwksLocation)
    if err != nil {
        msg := "cannot get the JWKS content for authentication"
        log.Error().Err(err).Msg(msg)
        panic(msg)
    }
    defer r.Body.Close()

    // Downloads the JSON Web Key Set
    body, err := ioutil.ReadAll(r.Body)
    if err != nil {
        msg := "cannot read the JWKS content for authentication"
        log.Error().Err(err).Msg(msg)
        panic(msg)
    }

    // Creates the AuthenticatorConfig structure
    config := middleware.AuthenticatorConfig{
        KeySetJSON: body,
        Issuer:     getRequiredEnv(AUTH_TOKEN_ISS),
    }

    log.Debug().
        Str("auth_token_iss", config.Issuer).
        Str("auth_jwks_location", jwksLocation).
        Msg("authenticator config loaded")

    return &config
}

Unit testing the changes in the main.go file

🏋️‍♀️ CHALLENGE: try to implement this by yourself before proceeding. Tip: you need to use the approach we used in the Library when using an external API.

The content added to the the main_test.go file is:

func TestNewAuthenticator(t *testing.T) {
    // arrange
    issuer := "https://cognito-idp.$AWS_REGION.amazonaws.com/$POOL_ID"
    sampleJwks := `
    {
        "keys": [{
            "kid": "1234example=",
            "alg": "RS256",
            "kty": "RSA",
            "e": "AQAB",
            "n": "1234567890",
            "use": "sig"
        }, {
            "kid": "5678example=",
            "alg": "RS256",
            "kty": "RSA",
            "e": "AQAB",
            "n": "987654321",
            "use": "sig"
        }]
    }`

    svr := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        fmt.Fprint(w, sampleJwks)
    }))

    os.Setenv(AUTH_TOKEN_ISS, issuer)
    os.Setenv(AUTH_JWKS_LOCATION, svr.URL)

    // act
    config := newAuthenticatorConfig()

    // assert
    assert.Equal(t, []byte(sampleJwks), config.KeySetJSON)
    assert.Equal(t, issuer, config.Issuer)
}

Manual testing

We first need to export the required environment variables:

export AUTH_JWKS_LOCATION=https://cognito-idp.$AWS_REGION.amazonaws.com/$POOL_ID/.well-known/jwks.json
export AUTH_TOKEN_ISS=https://cognito-idp.$AWS_REGION.amazonaws.com/$POOL_ID
export CURRCONV_API_KEY=your_currconf_api_key_here

Start the API:

go run main.go

And make the request:

http localhost:8080/v1/programming/uuid

You should now see an authentication error:

HTTP/1.1 401 Unauthorized
Content-Length: 28
Content-Type: application/json; charset=utf-8
Date: Fri, 07 Jan 2022 08:03:47 GMT

{
    "message": "Not authorized"
}

We need to generate a new token, like we did in the previous chapter:

http POST $TOKEN_ENDPOINT \
    Authorization:$AUTH_HEADER \
    Content-Type:application/x-www-form-urlencoded \
    --raw "grant_type=client_credentials&scope=https://learninggolang.com/all"

Use the access_token to call the API:

http localhost:8080/v1/programming/uuid Authentication:eyJraWQiOiJrb1wvR2owY1JrdFwvd2hQbm9Ne...

The result should be similar to:

HTTP/1.1 200 OK
Content-Length: 51
Content-Type: application/json; charset=utf-8
Date: Fri, 07 Jan 2022 08:09:25 GMT

{
    "uuid": "83f26b6a-203a-4ce6-b17e-78d994cb99d8"
}

Configure Kubernetes secrets to add the new environment variables

Go to the learning-go-api-iac repository.

First create a file named secrets.yaml with the following contents:

apiVersion: v1
kind: Secret
metadata:
  name: learning-go-api-secrets
type: Opaque
data:
  CURRCONV_API_KEY: <the value of the API key in base64>
  AUTH_JWKS_LOCATION: <the value of the AUTH_JWKS_LOCATION in base64>
  AUTH_TOKEN_ISS: <the value of the AUTH_TOKEN_ISS in base64>

Update the secrets running:

kubectl apply -f secrets.yaml -n learning-go-api

As the secrets.yaml file contains sensitive information, delete it:

rm secrets.yaml

Then add the new environment variables in the deployment.yaml file:

# ...
          env:
            - name: CURRCONV_API_KEY
              valueFrom:
                secretKeyRef:
                    name: learning-go-api-secrets
                    key: CURRCONV_API_KEY
            - name: AUTH_JWKS_LOCATION # new
              valueFrom:
                secretKeyRef:
                    name: learning-go-api-secrets
                    key: AUTH_JWKS_LOCATION
            - name: AUTH_TOKEN_ISS # new
              valueFrom:
                secretKeyRef:
                    name: learning-go-api-secrets
                    key: AUTH_TOKEN_ISS
# ...

In the Charts.yaml file, update the versions:

apiVersion: v2
name: learning-go-api
description: A Helm chart for Kubernetes
type: application
version: 0.1.1      # changed
appVersion: "0.0.8" # changed

Commit and push everything.

git add .
git commit -m "feat: add authentication to the api"
git push

Wrap up

Go back to the learning-go-api repository.

Commit and push everything. Create a new tag.

git add .
git commit -m "feat: add authentication to the api"
git push
git tag -a v0.0.9 -m "v0.0.9"
git push origin v0.0.9

After some minutes the k8s cluster should be updated.

Running:

kubectl -n flux-system get imagepolicies.image.toolkit.fluxcd.io

Should return the latest image:

NAME              LATESTIMAGE
learning-go-api   renato0307/learning-go-api:0.0.9

Check if the pods started up correctly:

kubectl -n learning-go-api get pods

The result should be a list of pods recently created (AGE column):

NAME                               READY   STATUS    RESTARTS   AGE
learning-go-api-6cc97b95cc-pmrpd   1/1     Running   0          1m24s
learning-go-api-6cc97b95cc-wb579   1/1     Running   0          1m33s

Execute the manual tests using the endpoint for the API running in the cluster to check if everything is working as expected.

Next

The next section is Add authorization to the API.