- crash dump recover does not log the call stack - impossible to debug
- sqs local & cloud
- cron local & cloud
- use in v3 and shiplogic
- config for local running - from cdk stuff...
- db connection in app + claims in app
- api-docs not yet working here - and need to provide HTML option or per-endpoint options at least
- log as JSON when not running local
- remove log clutter from API but allow switch on/off or dump on error using log sync...
- log as JSON - see if crash dump etc are clearly logged as fields, not as part of a string
# Later
- add path parameters, e.g. /user/{user_id} - make sure it works the same way as lambda does - but we're not using it at present, I assume because the old simple map[][] router did not support it.
- document with examples and templates
- scheduled tasks from the db or some other scheduler from AWS?
- clone db locally then run with local db not to mess up dev for others
- API test sequences configure & part of docs
# API
## Overview
API is a ```syncronous```[service](../service/README.md), meaning the user makes an HTTP(S) request and waits for the response. Handlers are associated with an HTTP method and path and should return ASAP. If there are any long running processes associated with your handler, they should be implemented as asynchronous services using [queues](../queues/README.md).
## Contents
*[Usage](#usage)
*[Examples](#examples)
## Usage
The main function of your API must:
1) Create the API and specify:
- the key used for request-id in HTTP headers,
- the HTTP routes (paths, methods and handlers)
Example:
api.New("uafrica-request-id", apiRoutes())
The routes are defined as a map of path with a map of method storing the handler, e.g.:
The handler function signature is discussed [here](#handlers).
2) Define Starters (optional)
See [Service Starters](../service/README.MD#define-starters)
3) Define Checks (optional)
See [Service Checks](../service/README.MD#define-checks)
4) Define Events Handlers (for testing only)
If the API makes async calls, i.e. if any of your handlers are sending queued events to SQS, then you could also define the event handlers in your API for local testing. When you do and you run locally, the SQS producer/consumer will be replaced by a in-memory implementation using Go channels. This means that your API will not send SQS event, but queue the processing internally, allowing you to do full end-to-end testing and even stepping through the async event handler code as part of your API debugger.
If you do not define the events, the API will still be able to send queued events, but they will always go to AWS SQS.
To define events, do:
.WithEvents(myEvents)
Where for example:
myEvents = map[string]interface{} {
"add-user": users.AddHandler,
"del-user": users.DelHandler,
}
4) Define a Local Port (for testing only)
You could define a local TCP port using a flag, which will allow you to run from the console using command line without using mage.
This feature is working but still under development, so rather not use it and run in mage where you have the correct environment setup defined in CDK.
When used, it runs with a normal golang HTTP server, not AWS Lambda. It also supports concurrent processing of multiple HTTP requests in one image, which AWS Lambda never does.
When the port is nil or has value 0, it will not apply and run as AWS Lambda.
So when mage runs your executable without the -port=xxx option, the local port will not be used and your API runs as an AWS Lambda function.
6) Run()
Now the service definition is complete, it must run.
This will run as AWS Lambda unless the local port is defined, then it will start to listen on the specified TCP port for HTTP traffic.
## Example
See [example1/api/api.go](../examples/core/api/main.go)
# Handlers
Each handler function referenced in your routes must have the following signature:
```
func MyHandler(
ctx api.Context,
params MyParams,
body MyBody,
) (
res MyRes,
err error,
) {
...
}
```
Notes:
*```params``` is required but may be empty struct{} for handlers that does not expect any URL parameters.
*```body``` may be omitted for handlers that does not process an HTTP body.
*```res``` may be omitted for handlers that does not respond with content.
The API do support legacy handlers also with the following signature. In this case, the handler has no context or other benefits from the framework. These handers should be phased out.
func (
ctx lambda.Context,
req events.APIGatewayProxyRequest,
) (
events.APIGatewayProxyResponse,
error,
) {
...
}
See the following regarding struct types used in handlers:
On top of above rules regarding input struct, the following applies specifically to API parameter structs.
Body struct are populated with ```json.Unmarshal()``` but params has a manual parsing from the URL that does not implement all types. In generally we use only strings and int64 types for fields in the params struct. This may change in future releases.
The json tag of a field must be used in the URL as param name.
Arrays of strings or int64 may also be used for parameters, e.g.:
type MyParams struct {
IDs []int64 `json:"ids"`
}
Then a URL may specify either:
*```?ids=[1,2,3]```, or
*```?ids=1&ids=2&ids=3```
Embedded structs may also be used, e.g. a generic struct for paging with Offset and Limit, embedded into a get struct that adds an ID, and embed that into a handler specified type with more parameters, as in this example:
type GenericPageParams struct {
Limit int64 `json:"limit"`
Offset int64 `json:"offset"`
}
type GenericGetParams struct {
ID int64 `json:"id"`
GenericPageParams
}
type MyGetParams struct {
GenericGetParams
Name string `json:"name"`
}
Call the embedded validation in the parent struct:
func (p MyGetParams) Validate() error {
if p.Name == "" { return errors.Errorf("missing name") }
In future, we could use a tag to automate embedded validation ...
## QueryStringParameters()
Some legacy functions operator from parameters in a map, like ```QueryStringParameters()``` returning ```map[string]string```.
That can still be used but is discorouged because:
* AWS API Gateway puts multiple params in another map - which you will miss when reading only that.
* Params structs parses both and supports array as multiple params or as CSV
* Params structs restrics the code from using adhoc params that are not documented anywhere
* Params structs enables us to generate documentation
* Params structs parses strings to int, meaning your code does not have to.
In future, we will extend the struct parsing to support many more features, including validation of values, out of the box, not requiring code in Validate() method. The pain for you is only to list all your params in a struct, then you will keep on getting benefits... avoid using maps!
If you still call functions that expects a map, you can convert your struct to a map using ```struct_utils.MapParams(myParamsStructValue)```.
## Errors
When a handler cannot deal with an error, it returns an error. The response value will be discarded, so the handler can just return an empty struct of the correct type.
The error could be any go error, which will result in an HTTP 500 Internal Server Error.
To return another HTTP error code, use:
return MyRes{}, errors.HTTP(code, err, message)
Note: The ```err != nil``` is required but the message may be "". If a message is added, it will wrap the error in another layer.
If the error is passed up through a few layers, and multiple HTTP codes are defined, the lowest code in the error stack will be returned to the user.
# Local Testing
When API is executed in local environment, with:
* env LOG_LEVEL=debug, and
* api.WithEvents(app.QueueRoutes())
Then any async events are sent into golang channels and processed locally, without delay. This allows you do debug and set break points in async services inside your api executable.
When executing the api from the console using a local port, the Lambda function is not used and instead the API runs in a normal http server on the specified port. Since it is not running with mage, it will load config.local.json to complete the environment normally defined in mage/cdk.
# API Documentation
API documentation will eventually be generated from doc tags in your structs. This is not yet complete...
# API Logger
API uses the go-utils/logger at the moment. Eventually it will be possible to use other loggers, and customise api logs. Audits can already be customised using ```api.WithAuditor()```.
By default, go-utils/logger writes JSON records. At the start of the example api main function the logger is configured to write console format and log at DEBUG level... more to come on this front too including triggers for full debug in production on selected handlers or specific events, and logging different levels for selected code packages to avoid log clutter without having to remove debug from code.
# Router Path Parameters
It is not yet possible to use path parameters, e.g. ```/user/{user_id}```.
# Automated Testing
Automated Testing is not yet part of the code base. It should be...
Queues are ```asyncronous```[services](../service/README.md), meaning a service request is fired into a queue and processed when it gets to the front of the queue. There is no response from the service, but it could fail and be scheduled for another attempt after an optional delay.
Asynchronous services should be used for any tasks that may take significant time to complete. The user could be offered API end-points to check on progress, get the result or cancel, as applicable.
Queues are implemented with AWS SQS in production but replaced with in-memory channels in Go when doing local testing of your [API](../api/README.md). It is also possible to use AWS SQS while doing local API testing.
## Contents
*[Usage](#usage)
*[Examples](#examples)
## Usage
The main function of your queue handler only runs in AWS SQS. For local testing, it is running inside the API handler. So you queue handler main function only creates an SQS service.
1) Create the SQS service and specify:
- the key used for request-id from event headers,
- the event names and handler functions.
Example:
sqs.New("uafrica-request-id", sqsRoutes())
The handler functions are defined as a map of event name and handler functions, e.g.:
func sqsRoutes() map[string]interface{} {
return map[string]interface{}{
"pay-invoice": invoices.Pay,
"send-email": emails.Send,
}
}
2) Define Starters (optional)
See [Service Starters](../service/README.MD#define-starters)
3) Define Checks (optional)
See [Service Checks](../service/README.MD#define-checks)
4) Run() or ProcessFile()
Now the service is defined it must Run() on a queue or one can test by processing one event from a file.
We call Run() in production, or ProcessFile() for local testing.
## Examples
See [example1/sqs/sqs.go](../examples/core/sqs/main.go)
# Handlers
Each handler function referenced in your routes must have the following signature:
```
func MyHandler(ctx queues.Context, body MyBody) (err error) {
...
}
```
Notes:
* The route name is the name specified in
```service.NewEvent(...).Type(<name>)```
* The ```body``` should have the same type you used elsewhere in
```service.NewEvent(...).Send(<type>)```
* The optional body type Validate() method will be called and must return nil before the handler will be called.
## Validate() error
If body struct type implements Validate() error methods, they will be called before the handler is called. If either returns an error, the HTTP call fails with BadRequest with the error in the content and the handler will not be called.
The Validate() method does not get a ctx, because it should only check the values inline and not do any db lookup or other service calls. Any such checks should be done inside the handler.
You Validate() method may take a pointer receiver and fill in default values. If it takes a value received, it cannot change the value. For example, in this Validate() we set default limit to 10:
Both params and body types must be structs with public fields and json tags, e.g.:
type MyParams struct {
ID int64 `json:"id"`
Offset int64 `json:"offset"`
Limit int64 `json:"limit"`
Name string `json:"name"`
}
The struct type is never marshaled to JSON by the framework, so omitempty has no effect.
Pointers may be used for optional items, or you may treat a zero value to indicate absence.
_Note: Body struct is populated with ```json.Unmarshal()``` but params has a manual parsing from the URL that does not implement all types. In generally we use strings and int64 for params. This may change in future releases._
Arrays may be used for parameters, e.g.:
type MyParams struct {
IDs []int64 `json:"ids"`
}
Then a URL may specify either:
*```?ids=[1,2,3]```, or
*```?ids=1&ids=2&ids=3```
You may include embedded structs in both params and body. Typical you may have a generic struct for paging with Offset and Limit, embed that into a get struct that adds an ID, and extend that for a handler to add more parameters, as in this example:
type GenericPageParams struct {
Limit int64 `json:"limit"`
Offset int64 `json:"offset"`
}
type GenericGetParams struct {
ID int64 `json:"id"`
GenericPageParams
}
type MyGetParams struct {
GenericGetParams
Name string `json:"name"`
}
You should then call the embedded validation, else it will not happen:
func (p MyGetParams) Validate() error {
if p.Name == "" { return errors.Errorf("missing name") }
The Validate() method is always called with a pointer to the struct. So if your Validate() method takes a pointer receiver, it can change values. If it takes value receiver it can't. It is up to you, but recommendation is to use a pointer receiver ONLY when you do change the value in some cases, e.g. setting defaults that are not zero.
The handler is always called with the struct values for params and body. The API creation in yuour main function will fail if you define any handler that takes a pointer type for either params of body.
That also means your handler should not really change the param/body values. It can, as the changes will apply to the rest of the handler, e.g. if you do some db lookups to fill in some blanks in the struct. Just note that both params and body values are discarded when the handler returns. The framework will therefore log the original values passed to your handler after validation.
## Errors
When a handler cannot deal with an error, it returns an error that will be logged.
# Async Event Retries
Async services can send the same event to be retried after a delay, although retries are also supported in AWS SQS when the handler failed. Sending a retry inside the code gives us more control over varying retry schedules for different error cases... not yet sure what SQS offers and local channels offers no delay at present.
Services are classified as API, SQS, CRON or ADHOC:
- API services are synchronous, i.e. the user waits for the response,
- SQS services are asynchronous, i.e. the request is queued and completes in good time without a response,
- CRON services are scheduled to run at regular intervals with no request nor response, and
- ADHOC services are executed on demand.
Package go-utils/services defines the common layer among all the above services, and then there are packages specific to each of them to extend the service as needed.
When you write and ```api```, you will import ```go-utils/api```.
For ```cron```, import ```go-utils/cron```.
For ```adhoc```, we have not yet created a package... TODO.
For ```sqs``` it is a bit more complicated because we have two implementations, allowing the use of AWS SQS in production or simulating queues using golang channels when you debug. Still, you will import from ```go-utils/queues```.
# Service Definition
The following can be added to any type of service during the service definition, typically in your main function.
## Starters
Each starter will be called at the start of request processing, before your handler is called, and must succeed. If any of them returns and error, the service fails and your handler is not called.
Starters are always called in the order they were defined.
Starter are typically used to ensure you have a database and/or redis connection.
The starter does not have to return a value, but must return err==nil.
If it does return a value, that value could be retrieve in a handler from the ctx using ctx.Get("db") which is the name specified in WithStarter().
Use type assert on the value to get the correct type returned from the starter, e.g.:
var dbConn *sql.Conn
{
db,_ := ctx.Get("db")
dbConn = db.(*sql.Conn)
}
result, err := dbConn.Query(...)
## Checks
Checks are similar to starter, but called after all started completed, also in the order defined.
Checks are typically used for applying rate limits or authorizastion.
Since the starters completed by the time the check is called, the check may rely on starter values, e.g. it may use a db connection created by a starter.
The only differences between service Starters and service Checks are:
* starters are always called before checks (even if defined after), and
* starters can return a value for handlers to retrieve which checks cannot do.
Example:
.WithCheck("claims", claims.ClaimsChecker{}).
.WithCheck("rate", rates.Limiter(10))
Each check implements service.ICheck and must return err==nil for processing to continue, or a descriptive reason for failing, e.g.:
return errors.Errorf("rate limited for user(%s)", username)
If your check is a pointer, the Check method gets a pointer receiver, and the check can set values in itself for subsequent calls, e.g. to count the nr of times it was called.
Note that your check only lives in this instance and if it does rate limiting, the value should NOT be stored locally but rather in a central service such as REDIS.
## Struct Types
Struct types used for input params/body must be structs with public fields and json tags, e.g.:
type MyParams struct {
ID int64 `json:"id"`
Offset int64 `json:"offset"`
Limit int64 `json:"limit"`
Name string `json:"name"`
}
The struct type is never marshaled to JSON by the framework, so omitempty has no effect.
Pointers may be used for optional items, or you may treat a zero value to indicate absence.
## Validate()
A ```Validate() error``` method is supported on all request and event structs such as API params and body and queued event body structs.
If the struct type implements a ```Validate() error``` method, it is called after populating the struct and applying the claim, before the handler is called. If it returns an error, the service fails. In API, this will result in HTTP status BadRequest with the error in the content and the handler will not be called. In SQS it will log the error.
The Validate() method does not get a ctx, because it should only check the values inline and not do any db lookup or other service calls. Any such checks should be done inside the handler.
When Validate() method takes a pointer receiver it could fill in default values. If it takes a value received, changes to the struct are lost after the call to Validate(). For example, in this Validate() a default of 10 is set in Limit:
When Validate() does not modify the struct, it should take a value receiver. There is no rule to enforce that, just good practice.
## Pointer or Value
The Validate() method is always called with a pointer to the struct. So if your Validate() method takes a pointer receiver, it can change values. If it takes value receiver it can't. It is up to you, but recommendation is to use a pointer receiver ONLY when you do change the value in some cases, e.g. setting defaults that are not zero.
The handler is always called with the struct values for params and body. The API creation in yuour main function will fail if you define any handler that takes a pointer type for either params of body.
That also means your handler should not really change the param/body values. It can, as the changes will apply to the rest of the handler, e.g. if you do some db lookups to fill in some blanks in the struct. Just note that both params and body values are discarded when the handler returns. The framework will therefore log the original values passed to your handler after validation.
# Claims
Claims identifies the user and typically consists of a user, account, role, permissions, etc...
Claims are populated by an service.Check added to your API and should be set in the service context using ```service.Context.ClaimSet()``` method.
Claim values may only be ```string``` or ```int64```. ```ClaimSet()``` will fail for other type of values.
Claim values are identified by a field name which must be a public field name in Go, e.g. ```"AccountID"``` rather than ```"account_id"```. Once again, ```ClaimSet()``` will fail if not.
If any of your param or body structs or sub-structs contains a field mathing a claim, they will be overwritten with the claim value before ```Validate()``` method is called. If your struct field match by name but use a type other than int64 or string, the handler will not be called with an error in claims.
So:
* Add claims to your API using ```.WithCheck("claim", myClaimChecker{})```
* In ```myClaimChecker.Check()```:
* Retrieve the claim, typically from a combination of:
* The context (e.g. AWS values)
* The HTTP header (e.g. some user id header)
* The HTTP URL params (e.g. a token string)
* The env (if you run in debug forcing some claim)
* ...
* Use your db/redis connections that was established in some ```api.WithStarter(...)``` to validate the claim or fail if not allowed
* Set the with calls to ```ctx.SetClaim(...)```
* Optionally, return a struct with all the claims, which handlers can retrieve using something like:
* All fields in params or body structs matching claim names will be overwritten by the time a handler is called.
# Audits
Audit records are written with:
* ctx.AuditChange(), or
* ctx.AuditWrite()
The AuditChange() method logs the changes between an original and new value.
The AuditWrite() logs all the data given to it.
A handler may write 0..N audit record, there is no check. In general, audits are written to capture changes, and when a handler changes multiple database records, they could all be audited.
# Sending Async Events
Events are sent for async processing with ```ctx.NewEvent()...Send()``` as in this example:
if _, err := service.NewEvent(ctx, "BILLING").
Type("provider-invoice").
RequestID(ctx.RequestID()).
Delay(time.Second * 5).
Send(Invoice{ID: invoice.ID}); err != nil {
ctx.Errorf("Failed to send event: %+v", err)
}
It is important to check the error, because it will fail if you specified invalid arguments or the producer fails to send for some reason such as authentication or the queue is not configured in the ENV etc...
The 2nd argument in NewEvent is the queue name. There must be an environment variable defined with that ```<name>_QUEUE_URL=...```.
The Delay() is optional and should only be used to delay processing of the event, e.g. to schedule a retry. The queue implementation may also impose restrictions on how long you can delay, e.g. AWS SQS may limits to 900 seconds.
