Search

This package is a wrapper around the AWS OpenSearch package to provide a consistent way to:

• log time series data (e.g. api-logs)
• store and search for documents

Time Series Logs

We use a time series for example to store API logs, writing a log record at the end of each API handler, with a start timestamp and duration as well as key values like HTTP method, path, request-id, URL parameters, response code etc.

The indexed logs are then mainly used from the OpenSearch dashboard to visualise the system usage and performance and we can setup alerts on conditions to detect failures and poor performance. One can also read the logs to asist trouble shooting.

The indexed logs can also be queried from our API to provide insights to account users, but the account users do not directly interact with OpenSearch.

Document Store and Search

We use a document store and search to provide quick text searches from the API. In this case a copy of a document (e.g. an order) is stored in OpenSearch when the order is created or updated. Updates overwrite the existing document, so there will only be one copy of each document.

When a user is looking for an order, the API provides an end-point to search orders e.g. for "lcd screen", then the API does an OpenSearch query in the orders index, get N results and then read those orders from the orders table in the database (not OpenSearch) and return those results to the user.

We therefore use OpenSearch only for searching and returning a list of document ids, then read the documents from the database. A document is typically an "order" but also anything else that we need to do free text searches on.