Skip to content
Snippets Groups Projects
Select Git revision
  • 7d54e7f76c62ebcf8687274b73439adaac141a63
  • main default protected
  • trading_hours
  • refactor_trading_hours
  • audit_cleaning_cater_for_non_struct_fields
  • remove-info-logs
  • sl-refactor
  • 18-use-scan-for-param-values
  • 17-order-search-results
  • 4-simplify-framework-2
  • 1-http-error
  • v1.297.0
  • v1.296.0
  • v1.295.0
  • v1.294.0
  • v1.293.0
  • v1.292.0
  • v1.291.0
  • v1.290.0
  • v1.289.0
  • v1.288.0
  • v1.287.0
  • v1.286.0
  • v1.285.0
  • v1.284.0
  • v1.283.0
  • v1.282.0
  • v1.281.0
  • v1.280.0
  • v1.279.0
  • v1.278.0
31 results

logs

user avatar
Jan Semmelink authored
7d54e7f7
History

Logs

This package provides functions to log API, SQS and CRON events, capturing for example an API method and path along with request and response details, or SQS event details along with the time it spent in the queue before being processed.

These logs are sent to a SQS queue. The handling of the queue event and capturing of the log, is done as SQS handlers, not as part of this package.

Do not confuse this with logger, which write free-format log entries to stdout.

API-LOGS

At the end of each API handler, an api-log is captured to describe the incoming API request and response. Also part of this log is a list of actions takens during the handler, including:

  • API calls made
  • SQS Events sent
  • Database Statements executed
  • Time slept

Each of those has a start/end time and duration, and they are summed and it should add up to most of the API-Log total duration. If there is a big difference between the summed time and the total duration, then we are doing something that takes time that we are not monitoring which should be investigated. The total time spent sleeping, waiting for db calls, waiting for outgoing API calls, is logged in the API log. This can be logged and broken down per path and method to see where the API is spending most time, and that could be investigated and optimised to improvie performance.

SQS-LOGS

SQS logs are written at the end of a SQS event handler, similar to API logs. Since SQS is used to write API logs, those handlers should not be logged, as it will create a circular infinite queue ramping up huge bills. To be safe, SQS logs are therefore DISABLED by default. It should only be enabled for things like provider rate requests or any SQS handler that is part of the functionality of the system doing async work, not handlers that are part of the infrastructure.

SQS log will also write to the API_LOGS queue and the same index in OpenSearch (can review and change this in future) It logs with method "SQS" and path is the message type. That means we can log durations and through put in the same way and on the same graph as API when needed It also captures the actions taken as part of the handler, in the same way it is captured for API.

So when one finds some action takes too long in API, and move it to an SQS handler, the change will be visibile on the dashboard and indicate the improvement or not if your change did not have the desired effect.

That it is idea.

We can easily disable SQS logs and we can easily move it to another index in OpenSearch if necessary. Will have to try it for a while an see if it is useful in the current form or not.

CRON-LOGS

In the same way we log API/SQS, it will be useful to monitor crons with a bit of output, e.g. nr of items deleted by a janitor etc. One can get that currently from cloud watch if the logs are not disabled, and CloudWatch should ideally not be source of metrics, but that is currently the case, so not changing it yet.