Compare commits
22 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
d03271d128 | ||
|
|
0560b98de4 | ||
|
|
ca87547430 | ||
|
|
e214d56af1 | ||
|
|
8997eeef05 | ||
|
|
5e00752c5a | ||
|
|
f9d132c369 | ||
|
|
ca977fefa8 | ||
|
|
d07d3434a6 | ||
|
|
2131fa4412 | ||
|
|
81aeb7a48e | ||
|
|
eaf395738d | ||
|
|
f6f1ecf623 | ||
|
|
177081cf54 | ||
|
|
651bfcba22 | ||
|
|
3cd1953c6c | ||
|
|
9dd4e7047d | ||
|
|
067ab78666 | ||
|
|
28acaeb067 | ||
|
|
749aeb9e42 | ||
|
|
8e02572880 | ||
|
|
1f6f0ce426 |
BIN
.github/assets/dark-mode.png
vendored
Normal file
BIN
.github/assets/dark-mode.png
vendored
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 38 KiB |
40
README.md
40
README.md
@@ -10,11 +10,20 @@
|
||||
Gatus is a health dashboard that gives you the ability to monitor your services using HTTP, ICMP, TCP, and even DNS
|
||||
queries as well as evaluate the result of said queries by using a list of conditions on values like the status code,
|
||||
the response time, the certificate expiration, the body and many others. The icing on top is that each of these health
|
||||
checks can be paired with alerting via Slack, PagerDuty and even Twilio.
|
||||
checks can be paired with alerting via Slack, PagerDuty, Discord and even Twilio.
|
||||
|
||||
I personally deploy it in my Kubernetes cluster and let it monitor the status of my
|
||||
core applications: https://status.twinnation.org/
|
||||
|
||||
<details>
|
||||
<summary><b>Quick start</b></summary>
|
||||
|
||||
```
|
||||
docker run -p 8080:8080 --name gatus twinproduction/gatus
|
||||
```
|
||||
For more details, see [Usage](#usage)
|
||||
</details>
|
||||
|
||||
|
||||
## Table of Contents
|
||||
|
||||
@@ -47,6 +56,7 @@ core applications: https://status.twinnation.org/
|
||||
- [Monitoring a TCP service](#monitoring-a-tcp-service)
|
||||
- [Monitoring a service using ICMP](#monitoring-a-service-using-icmp)
|
||||
- [Monitoring a service using DNS queries](#monitoring-a-service-using-dns-queries)
|
||||
- [Monitoring a service using STARTTLS](#monitoring-a-service-using-starttls)
|
||||
- [Basic authentication](#basic-authentication)
|
||||
- [disable-monitoring-lock](#disable-monitoring-lock)
|
||||
- [Reloading configuration on the fly](#reloading-configuration-on-the-fly)
|
||||
@@ -77,6 +87,8 @@ fixing the issue before they even know about it.
|
||||
|
||||
## Features
|
||||
|
||||

|
||||
|
||||
The main features of Gatus are:
|
||||
- **Highly flexible health check conditions**: While checking the response status may be enough for some use cases, Gatus goes much further and allows you to add conditions on the response time, the response body and even the IP address.
|
||||
- **Ability to use Gatus for user acceptance tests**: Thanks to the point above, you can leverage this application to create automated user acceptance tests.
|
||||
@@ -85,7 +97,6 @@ The main features of Gatus are:
|
||||
- **Metrics**
|
||||
- **Low resource consumption**: As with most Go applications, the resource footprint that this application requires is negligibly small.
|
||||
- **GitHub uptime badges**:   
|
||||
- **Service auto discovery in Kubernetes** (ALPHA)
|
||||
|
||||
|
||||
## Usage
|
||||
@@ -119,6 +130,8 @@ This example would look like this:
|
||||
|
||||
Note that you can also use environment variables in the configuration file (e.g. `$DOMAIN`, `${DOMAIN}`)
|
||||
|
||||
If you want to test it locally, see [Docker](#docker).
|
||||
|
||||
|
||||
## Configuration
|
||||
|
||||
@@ -629,8 +642,13 @@ See [example/kubernetes-with-auto-discovery](example/kubernetes-with-auto-discov
|
||||
|
||||
## Docker
|
||||
|
||||
To run Gatus locally with Docker:
|
||||
```
|
||||
docker run -p 8080:8080 --name gatus twinproduction/gatus
|
||||
```
|
||||
|
||||
Other than using one of the examples provided in the `examples` folder, you can also try it out locally by
|
||||
creating a configuration file - we'll call it `config.yaml` for this example - and running the following
|
||||
creating a configuration file, we'll call it `config.yaml` for this example, and running the following
|
||||
command:
|
||||
```
|
||||
docker run -p 8080:8080 --mount type=bind,source="$(pwd)"/config.yaml,target=/config/config.yaml --name gatus twinproduction/gatus
|
||||
@@ -792,10 +810,24 @@ There are two placeholders that can be used in the conditions for services of ty
|
||||
`NOERROR`, `FORMERR`, `SERVFAIL`, `NXDOMAIN`, etc.
|
||||
|
||||
|
||||
### Monitoring a service using STARTTLS
|
||||
|
||||
If you have an email server that you want to ensure there are no problems with, monitoring it through STARTTLS
|
||||
will serve as a good initial indicator:
|
||||
```yaml
|
||||
services:
|
||||
- name: starttls-smtp-example
|
||||
url: "starttls://smtp.gmail.com:587"
|
||||
interval: 30m
|
||||
conditions:
|
||||
- "[CONNECTED] == true"
|
||||
- "[CERTIFICATE_EXPIRATION] > 48h"
|
||||
```
|
||||
|
||||
|
||||
### Basic authentication
|
||||
|
||||
You can require Basic authentication by leveraging the `security.basic` configuration:
|
||||
|
||||
```yaml
|
||||
security:
|
||||
basic:
|
||||
|
||||
@@ -2,10 +2,14 @@ package client
|
||||
|
||||
import (
|
||||
"crypto/tls"
|
||||
"crypto/x509"
|
||||
"errors"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/smtp"
|
||||
"os"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/go-ping/ping"
|
||||
@@ -74,6 +78,31 @@ func CanCreateTCPConnection(address string) bool {
|
||||
return true
|
||||
}
|
||||
|
||||
// CanPerformStartTLS checks whether a connection can be established to an address using the STARTTLS protocol
|
||||
func CanPerformStartTLS(address string, insecure bool) (connected bool, certificate *x509.Certificate, err error) {
|
||||
hostAndPort := strings.Split(address, ":")
|
||||
if len(hostAndPort) != 2 {
|
||||
return false, nil, errors.New("invalid address for starttls, format must be host:port")
|
||||
}
|
||||
smtpClient, err := smtp.Dial(address)
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
err = smtpClient.StartTLS(&tls.Config{
|
||||
InsecureSkipVerify: insecure,
|
||||
ServerName: hostAndPort[0],
|
||||
})
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
if state, ok := smtpClient.TLSConnectionState(); ok {
|
||||
certificate = state.PeerCertificates[0]
|
||||
} else {
|
||||
return false, nil, errors.New("could not get TLS connection state")
|
||||
}
|
||||
return true, certificate, nil
|
||||
}
|
||||
|
||||
// Ping checks if an address can be pinged and returns the round-trip time if the address can be pinged
|
||||
//
|
||||
// Note that this function takes at least 100ms, even if the address is 127.0.0.1
|
||||
|
||||
@@ -49,3 +49,53 @@ func TestPing(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestCanPerformStartTLS(t *testing.T) {
|
||||
type args struct {
|
||||
address string
|
||||
insecure bool
|
||||
}
|
||||
tests := []struct {
|
||||
name string
|
||||
args args
|
||||
wantConnected bool
|
||||
wantErr bool
|
||||
}{
|
||||
{
|
||||
name: "invalid address",
|
||||
args: args{
|
||||
address: "test",
|
||||
},
|
||||
wantConnected: false,
|
||||
wantErr: true,
|
||||
},
|
||||
{
|
||||
name: "error dial",
|
||||
args: args{
|
||||
address: "test:1234",
|
||||
},
|
||||
wantConnected: false,
|
||||
wantErr: true,
|
||||
},
|
||||
{
|
||||
name: "valid starttls",
|
||||
args: args{
|
||||
address: "smtp.gmail.com:587",
|
||||
},
|
||||
wantConnected: true,
|
||||
wantErr: false,
|
||||
},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
connected, _, err := CanPerformStartTLS(tt.args.address, tt.args.insecure)
|
||||
if (err != nil) != tt.wantErr {
|
||||
t.Errorf("CanPerformStartTLS() err=%v, wantErr=%v", err, tt.wantErr)
|
||||
return
|
||||
}
|
||||
if connected != tt.wantConnected {
|
||||
t.Errorf("CanPerformStartTLS() connected=%v, wantConnected=%v", connected, tt.wantConnected)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
@@ -126,7 +126,7 @@ func (c Condition) evaluate(result *Result) bool {
|
||||
conditionToDisplay = prettifyNumericalParameters(parameters, resolvedParameters, "<")
|
||||
}
|
||||
} else {
|
||||
result.Errors = append(result.Errors, fmt.Sprintf("invalid condition '%s' has been provided", condition))
|
||||
result.AddError(fmt.Sprintf("invalid condition '%s' has been provided", condition))
|
||||
return false
|
||||
}
|
||||
if !success {
|
||||
@@ -242,7 +242,7 @@ func sanitizeAndResolve(elements []string, result *Result) ([]string, []string)
|
||||
} else {
|
||||
if err != nil {
|
||||
if err.Error() != "unexpected end of JSON input" {
|
||||
result.Errors = append(result.Errors, err.Error())
|
||||
result.AddError(err.Error())
|
||||
}
|
||||
if checkingForLength {
|
||||
element = LengthFunctionPrefix + element + FunctionSuffix + " " + InvalidConditionElementSuffix
|
||||
|
||||
@@ -38,6 +38,15 @@ func BenchmarkCondition_evaluateWithBodyStringFailure(b *testing.B) {
|
||||
b.ReportAllocs()
|
||||
}
|
||||
|
||||
func BenchmarkCondition_evaluateWithBodyStringFailureInvalidPath(b *testing.B) {
|
||||
condition := Condition("[BODY].user.name == bob.doe")
|
||||
for n := 0; n < b.N; n++ {
|
||||
result := &Result{body: []byte("{\"name\": \"bob.doe\"}")}
|
||||
condition.evaluate(result)
|
||||
}
|
||||
b.ReportAllocs()
|
||||
}
|
||||
|
||||
func BenchmarkCondition_evaluateWithBodyStringLen(b *testing.B) {
|
||||
condition := Condition("len([BODY].name) == 8")
|
||||
for n := 0; n < b.N; n++ {
|
||||
|
||||
@@ -52,7 +52,7 @@ func (d *DNS) query(url string, result *Result) {
|
||||
m.SetQuestion(d.QueryName, queryType)
|
||||
r, _, err := c.Exchange(m, url)
|
||||
if err != nil {
|
||||
result.Errors = append(result.Errors, err.Error())
|
||||
result.AddError(err.Error())
|
||||
return
|
||||
}
|
||||
result.Connected = true
|
||||
|
||||
@@ -25,7 +25,7 @@ type Result struct {
|
||||
Duration time.Duration `json:"duration"`
|
||||
|
||||
// Errors encountered during the evaluation of the service's health
|
||||
Errors []string `json:"errors"`
|
||||
Errors []string `json:"errors"` // XXX: find a way to filter out duplicate errors
|
||||
|
||||
// ConditionResults results of the service's conditions
|
||||
ConditionResults []*ConditionResult `json:"conditionResults"`
|
||||
@@ -46,3 +46,15 @@ type Result struct {
|
||||
// and sets it to nil after the evaluation has been completed.
|
||||
body []byte
|
||||
}
|
||||
|
||||
// AddError adds an error to the result's list of errors.
|
||||
// It also ensures that there are no duplicates.
|
||||
func (r *Result) AddError(error string) {
|
||||
for _, resultError := range r.Errors {
|
||||
if resultError == error {
|
||||
// If the error already exists, don't add it
|
||||
return
|
||||
}
|
||||
}
|
||||
r.Errors = append(r.Errors, error)
|
||||
}
|
||||
|
||||
21
core/result_test.go
Normal file
21
core/result_test.go
Normal file
@@ -0,0 +1,21 @@
|
||||
package core
|
||||
|
||||
import (
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestResult_AddError(t *testing.T) {
|
||||
result := &Result{}
|
||||
result.AddError("potato")
|
||||
if len(result.Errors) != 1 {
|
||||
t.Error("should've had 1 error")
|
||||
}
|
||||
result.AddError("potato")
|
||||
if len(result.Errors) != 1 {
|
||||
t.Error("should've still had 1 error, because a duplicate error was added")
|
||||
}
|
||||
result.AddError("tomato")
|
||||
if len(result.Errors) != 2 {
|
||||
t.Error("should've had 2 error")
|
||||
}
|
||||
}
|
||||
@@ -2,6 +2,7 @@ package core
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"crypto/x509"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"io/ioutil"
|
||||
@@ -155,35 +156,20 @@ func (service *Service) EvaluateHealth() *Result {
|
||||
return result
|
||||
}
|
||||
|
||||
// GetAlertsTriggered returns a slice of alerts that have been triggered
|
||||
func (service *Service) GetAlertsTriggered() []alert.Alert {
|
||||
var alerts []alert.Alert
|
||||
if service.NumberOfFailuresInARow == 0 {
|
||||
return alerts
|
||||
}
|
||||
for _, alert := range service.Alerts {
|
||||
if alert.IsEnabled() && alert.FailureThreshold == service.NumberOfFailuresInARow {
|
||||
alerts = append(alerts, *alert)
|
||||
continue
|
||||
}
|
||||
}
|
||||
return alerts
|
||||
}
|
||||
|
||||
func (service *Service) getIP(result *Result) {
|
||||
if service.DNS != nil {
|
||||
result.Hostname = strings.TrimSuffix(service.URL, ":53")
|
||||
} else {
|
||||
urlObject, err := url.Parse(service.URL)
|
||||
if err != nil {
|
||||
result.Errors = append(result.Errors, err.Error())
|
||||
result.AddError(err.Error())
|
||||
return
|
||||
}
|
||||
result.Hostname = urlObject.Hostname()
|
||||
}
|
||||
ips, err := net.LookupIP(result.Hostname)
|
||||
if err != nil {
|
||||
result.Errors = append(result.Errors, err.Error())
|
||||
result.AddError(err.Error())
|
||||
return
|
||||
}
|
||||
result.IP = ips[0].String()
|
||||
@@ -193,10 +179,12 @@ func (service *Service) call(result *Result) {
|
||||
var request *http.Request
|
||||
var response *http.Response
|
||||
var err error
|
||||
var certificate *x509.Certificate
|
||||
isServiceDNS := service.DNS != nil
|
||||
isServiceTCP := strings.HasPrefix(service.URL, "tcp://")
|
||||
isServiceICMP := strings.HasPrefix(service.URL, "icmp://")
|
||||
isServiceHTTP := !isServiceDNS && !isServiceTCP && !isServiceICMP
|
||||
isServiceStartTLS := strings.HasPrefix(service.URL, "starttls://")
|
||||
isServiceHTTP := !isServiceDNS && !isServiceTCP && !isServiceICMP && !isServiceStartTLS
|
||||
if isServiceHTTP {
|
||||
request = service.buildHTTPRequest()
|
||||
}
|
||||
@@ -204,6 +192,14 @@ func (service *Service) call(result *Result) {
|
||||
if isServiceDNS {
|
||||
service.DNS.query(service.URL, result)
|
||||
result.Duration = time.Since(startTime)
|
||||
} else if isServiceStartTLS {
|
||||
result.Connected, certificate, err = client.CanPerformStartTLS(strings.TrimPrefix(service.URL, "starttls://"), service.Insecure)
|
||||
if err != nil {
|
||||
result.AddError(err.Error())
|
||||
return
|
||||
}
|
||||
result.Duration = time.Since(startTime)
|
||||
result.CertificateExpiration = time.Until(certificate.NotAfter)
|
||||
} else if isServiceTCP {
|
||||
result.Connected = client.CanCreateTCPConnection(strings.TrimPrefix(service.URL, "tcp://"))
|
||||
result.Duration = time.Since(startTime)
|
||||
@@ -213,12 +209,12 @@ func (service *Service) call(result *Result) {
|
||||
response, err = client.GetHTTPClient(service.Insecure).Do(request)
|
||||
result.Duration = time.Since(startTime)
|
||||
if err != nil {
|
||||
result.Errors = append(result.Errors, err.Error())
|
||||
result.AddError(err.Error())
|
||||
return
|
||||
}
|
||||
defer response.Body.Close()
|
||||
if response.TLS != nil && len(response.TLS.PeerCertificates) > 0 {
|
||||
certificate := response.TLS.PeerCertificates[0]
|
||||
certificate = response.TLS.PeerCertificates[0]
|
||||
result.CertificateExpiration = time.Until(certificate.NotAfter)
|
||||
}
|
||||
result.HTTPStatus = response.StatusCode
|
||||
@@ -227,7 +223,7 @@ func (service *Service) call(result *Result) {
|
||||
if service.needsToReadBody() {
|
||||
result.body, err = ioutil.ReadAll(response.Body)
|
||||
if err != nil {
|
||||
result.Errors = append(result.Errors, err.Error())
|
||||
result.AddError(err.Error())
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -102,31 +102,6 @@ func TestService_ValidateAndSetDefaultsWithDNS(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestService_GetAlertsTriggered(t *testing.T) {
|
||||
condition := Condition("[STATUS] == 200")
|
||||
enabled := true
|
||||
service := Service{
|
||||
Name: "twinnation-health",
|
||||
URL: "https://twinnation.org/health",
|
||||
Conditions: []*Condition{&condition},
|
||||
Alerts: []*alert.Alert{{Type: alert.TypePagerDuty, Enabled: &enabled}},
|
||||
}
|
||||
service.ValidateAndSetDefaults()
|
||||
if service.NumberOfFailuresInARow != 0 {
|
||||
t.Error("Service.NumberOfFailuresInARow should start with 0")
|
||||
}
|
||||
if service.NumberOfSuccessesInARow != 0 {
|
||||
t.Error("Service.NumberOfSuccessesInARow should start with 0")
|
||||
}
|
||||
if len(service.GetAlertsTriggered()) > 0 {
|
||||
t.Error("No alerts should've been triggered, because service.NumberOfFailuresInARow is 0, which is below the failure threshold")
|
||||
}
|
||||
service.NumberOfFailuresInARow = service.Alerts[0].FailureThreshold
|
||||
if len(service.GetAlertsTriggered()) != 1 {
|
||||
t.Error("Alert should've been triggered")
|
||||
}
|
||||
}
|
||||
|
||||
func TestService_buildHTTPRequest(t *testing.T) {
|
||||
condition := Condition("[STATUS] == 200")
|
||||
service := Service{
|
||||
|
||||
@@ -1,8 +1,8 @@
|
||||
version: "3.8"
|
||||
services:
|
||||
gatus:
|
||||
image: twinproduction/gatus:latest
|
||||
ports:
|
||||
- 8080:8080
|
||||
volumes:
|
||||
- ./config.yaml:/config/config.yaml
|
||||
gatus:
|
||||
image: twinproduction/gatus:latest
|
||||
ports:
|
||||
- 8080:8080
|
||||
volumes:
|
||||
- ./config.yaml:/config/config.yaml
|
||||
2
go.mod
2
go.mod
@@ -4,7 +4,7 @@ go 1.16
|
||||
|
||||
require (
|
||||
cloud.google.com/go v0.74.0 // indirect
|
||||
github.com/TwinProduction/gocache v1.2.1
|
||||
github.com/TwinProduction/gocache v1.2.3
|
||||
github.com/TwinProduction/health v1.0.0
|
||||
github.com/go-ping/ping v0.0.0-20201115131931-3300c582a663
|
||||
github.com/google/gofuzz v1.2.0 // indirect
|
||||
|
||||
4
go.sum
4
go.sum
@@ -49,8 +49,8 @@ github.com/PuerkitoBio/purell v1.0.0/go.mod h1:c11w/QuzBsJSee3cPx9rAFu61PvFxuPbt
|
||||
github.com/PuerkitoBio/urlesc v0.0.0-20160726150825-5bd2802263f2/go.mod h1:uGdkoq3SwY9Y+13GIhn11/XLaGBb4BfwItxLd5jeuXE=
|
||||
github.com/Shopify/sarama v1.19.0/go.mod h1:FVkBWblsNy7DGZRfXLU0O9RCGt5g3g3yEuWXgklEdEo=
|
||||
github.com/Shopify/toxiproxy v2.1.4+incompatible/go.mod h1:OXgGpZ6Cli1/URJOF1DMxUHB2q5Ap20/P/eIdh4G0pI=
|
||||
github.com/TwinProduction/gocache v1.2.1 h1:NAdMwO9SQEZFmX69YWx6fzhwb6fHakkLri0451c+V1w=
|
||||
github.com/TwinProduction/gocache v1.2.1/go.mod h1:6zkBoLjrFLkIISwkZTgLy67qliCGSon1xpORM4Ri5HM=
|
||||
github.com/TwinProduction/gocache v1.2.3 h1:4wFNih4CemUX+A99Gk/EsaU0SXSNZV42Ve77v7/7ToY=
|
||||
github.com/TwinProduction/gocache v1.2.3/go.mod h1:Yj2daITit8TTBgiOpc26XCDSbg9xcFskUilHj9u3Mh8=
|
||||
github.com/TwinProduction/health v1.0.0 h1:TVyYTAORQQZ8LaptX8jCHZRCGCAO6e+oJx19BUIzQYY=
|
||||
github.com/TwinProduction/health v1.0.0/go.mod h1:ys4mYKUeEfYrWmkm60xLtPjTuLIEDQNBZaTZvenLG1c=
|
||||
github.com/VividCortex/gohistogram v1.0.0/go.mod h1:Pf5mBqqDxYaXu3hDrrU+w6nw50o/4+TcAqDqk/vUH7g=
|
||||
|
||||
@@ -25,6 +25,9 @@ func walk(path string, object interface{}) (string, int, error) {
|
||||
case map[string]interface{}:
|
||||
return walk(strings.Replace(path, fmt.Sprintf("%s.", currentKey), "", 1), value)
|
||||
case string:
|
||||
if len(keys) > 1 {
|
||||
return "", 0, fmt.Errorf("couldn't walk through '%s', because '%s' was a string instead of an object", keys[1], currentKey)
|
||||
}
|
||||
return value, len(value), nil
|
||||
case []interface{}:
|
||||
return fmt.Sprintf("%v", value), len(value), nil
|
||||
|
||||
@@ -1,180 +1,148 @@
|
||||
package jsonpath
|
||||
|
||||
import "testing"
|
||||
import (
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestEval(t *testing.T) {
|
||||
path := "simple"
|
||||
data := `{"simple": "value"}`
|
||||
|
||||
expectedOutput := "value"
|
||||
|
||||
output, outputLength, err := Eval(path, []byte(data))
|
||||
if err != nil {
|
||||
t.Error("Didn't expect any error, but got", err)
|
||||
type Scenario struct {
|
||||
Name string
|
||||
Path string
|
||||
Data string
|
||||
ExpectedOutput string
|
||||
ExpectedOutputLength int
|
||||
ExpectedError bool
|
||||
}
|
||||
if outputLength != len(expectedOutput) {
|
||||
t.Errorf("Expected output length to be %v, but was %v", len(expectedOutput), outputLength)
|
||||
scenarios := []Scenario{
|
||||
{
|
||||
Name: "simple",
|
||||
Path: "key",
|
||||
Data: `{"key": "value"}`,
|
||||
ExpectedOutput: "value",
|
||||
ExpectedOutputLength: 5,
|
||||
ExpectedError: false,
|
||||
},
|
||||
{
|
||||
Name: "simple-with-invalid-data",
|
||||
Path: "key",
|
||||
Data: "invalid data",
|
||||
ExpectedOutput: "",
|
||||
ExpectedOutputLength: 0,
|
||||
ExpectedError: true,
|
||||
},
|
||||
{
|
||||
Name: "invalid-path",
|
||||
Path: "key",
|
||||
Data: `{}`,
|
||||
ExpectedOutput: "",
|
||||
ExpectedOutputLength: 0,
|
||||
ExpectedError: true,
|
||||
},
|
||||
{
|
||||
Name: "long-simple-walk",
|
||||
Path: "long.simple.walk",
|
||||
Data: `{"long": {"simple": {"walk": "value"}}}`,
|
||||
ExpectedOutput: "value",
|
||||
ExpectedOutputLength: 5,
|
||||
ExpectedError: false,
|
||||
},
|
||||
{
|
||||
Name: "array-of-maps",
|
||||
Path: "ids[1].id",
|
||||
Data: `{"ids": [{"id": 1}, {"id": 2}]}`,
|
||||
ExpectedOutput: "2",
|
||||
ExpectedOutputLength: 1,
|
||||
ExpectedError: false,
|
||||
},
|
||||
{
|
||||
Name: "array-of-values",
|
||||
Path: "ids[0]",
|
||||
Data: `{"ids": [1, 2]}`,
|
||||
ExpectedOutput: "1",
|
||||
ExpectedOutputLength: 1,
|
||||
ExpectedError: false,
|
||||
},
|
||||
{
|
||||
Name: "array-of-values-and-invalid-index",
|
||||
Path: "ids[wat]",
|
||||
Data: `{"ids": [1, 2]}`,
|
||||
ExpectedOutput: "",
|
||||
ExpectedOutputLength: 0,
|
||||
ExpectedError: true,
|
||||
},
|
||||
{
|
||||
Name: "array-of-values-at-root",
|
||||
Path: "[1]",
|
||||
Data: `[1, 2]`,
|
||||
ExpectedOutput: "2",
|
||||
ExpectedOutputLength: 1,
|
||||
ExpectedError: false,
|
||||
},
|
||||
{
|
||||
Name: "array-of-maps-at-root",
|
||||
Path: "[0].id",
|
||||
Data: `[{"id": 1}, {"id": 2}]`,
|
||||
ExpectedOutput: "1",
|
||||
ExpectedOutputLength: 1,
|
||||
ExpectedError: false,
|
||||
},
|
||||
{
|
||||
Name: "array-of-maps-at-root-and-invalid-index",
|
||||
Path: "[5].id",
|
||||
Data: `[{"id": 1}, {"id": 2}]`,
|
||||
ExpectedOutput: "",
|
||||
ExpectedOutputLength: 0,
|
||||
ExpectedError: true,
|
||||
},
|
||||
{
|
||||
Name: "long-walk-and-array",
|
||||
Path: "data.ids[0].id",
|
||||
Data: `{"data": {"ids": [{"id": 1}, {"id": 2}, {"id": 3}]}}`,
|
||||
ExpectedOutput: "1",
|
||||
ExpectedOutputLength: 1,
|
||||
ExpectedError: false,
|
||||
},
|
||||
{
|
||||
Name: "nested-array",
|
||||
Path: "[3][2]",
|
||||
Data: `[[1, 2], [3, 4], [], [5, 6, 7]]`,
|
||||
ExpectedOutput: "7",
|
||||
ExpectedOutputLength: 1,
|
||||
ExpectedError: false,
|
||||
},
|
||||
{
|
||||
Name: "map-of-nested-arrays",
|
||||
Path: "data[1][1]",
|
||||
Data: `{"data": [["a", "b", "c"], ["d", "eeeee", "f"]]}`,
|
||||
ExpectedOutput: "eeeee",
|
||||
ExpectedOutputLength: 5,
|
||||
ExpectedError: false,
|
||||
},
|
||||
{
|
||||
Name: "partially-invalid-path-issue122",
|
||||
Path: "data.name.invalid",
|
||||
Data: `{"data": {"name": "john"}}`,
|
||||
ExpectedOutput: "",
|
||||
ExpectedOutputLength: 0,
|
||||
ExpectedError: true,
|
||||
},
|
||||
}
|
||||
if output != expectedOutput {
|
||||
t.Errorf("Expected output to be %v, but was %v", expectedOutput, output)
|
||||
}
|
||||
}
|
||||
|
||||
func TestEvalWithInvalidData(t *testing.T) {
|
||||
path := "simple"
|
||||
data := `invalid data`
|
||||
_, _, err := Eval(path, []byte(data))
|
||||
if err == nil {
|
||||
t.Error("expected an error")
|
||||
}
|
||||
}
|
||||
|
||||
func TestEvalWithInvalidPath(t *testing.T) {
|
||||
path := "errors"
|
||||
data := `{}`
|
||||
_, _, err := Eval(path, []byte(data))
|
||||
if err == nil {
|
||||
t.Error("Expected error, but got", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestEvalWithLongSimpleWalk(t *testing.T) {
|
||||
path := "long.simple.walk"
|
||||
data := `{"long": {"simple": {"walk": "value"}}}`
|
||||
|
||||
expectedOutput := "value"
|
||||
|
||||
output, _, err := Eval(path, []byte(data))
|
||||
if err != nil {
|
||||
t.Error("Didn't expect any error, but got", err)
|
||||
}
|
||||
if output != expectedOutput {
|
||||
t.Errorf("Expected output to be %v, but was %v", expectedOutput, output)
|
||||
}
|
||||
}
|
||||
|
||||
func TestEvalWithArrayOfMaps(t *testing.T) {
|
||||
path := "ids[1].id"
|
||||
data := `{"ids": [{"id": 1}, {"id": 2}]}`
|
||||
|
||||
expectedOutput := "2"
|
||||
|
||||
output, _, err := Eval(path, []byte(data))
|
||||
if err != nil {
|
||||
t.Error("Didn't expect any error, but got", err)
|
||||
}
|
||||
|
||||
if output != expectedOutput {
|
||||
t.Errorf("Expected output to be %v, but was %v", expectedOutput, output)
|
||||
}
|
||||
}
|
||||
|
||||
func TestEvalWithArrayOfValues(t *testing.T) {
|
||||
path := "ids[0]"
|
||||
data := `{"ids": [1, 2]}`
|
||||
|
||||
expectedOutput := "1"
|
||||
|
||||
output, _, err := Eval(path, []byte(data))
|
||||
if err != nil {
|
||||
t.Error("Didn't expect any error, but got", err)
|
||||
}
|
||||
if output != expectedOutput {
|
||||
t.Errorf("Expected output to be %v, but was %v", expectedOutput, output)
|
||||
}
|
||||
}
|
||||
|
||||
func TestEvalWithArrayOfValuesAndInvalidIndex(t *testing.T) {
|
||||
path := "ids[wat]"
|
||||
data := `{"ids": [1, 2]}`
|
||||
|
||||
_, _, err := Eval(path, []byte(data))
|
||||
if err == nil {
|
||||
t.Error("Expected an error")
|
||||
}
|
||||
}
|
||||
|
||||
func TestEvalWithRootArrayOfValues(t *testing.T) {
|
||||
path := "[1]"
|
||||
data := `[1, 2]`
|
||||
|
||||
expectedOutput := "2"
|
||||
|
||||
output, _, err := Eval(path, []byte(data))
|
||||
if err != nil {
|
||||
t.Error("Didn't expect any error, but got", err)
|
||||
}
|
||||
if output != expectedOutput {
|
||||
t.Errorf("Expected output to be %v, but was %v", expectedOutput, output)
|
||||
}
|
||||
}
|
||||
|
||||
func TestEvalWithRootArrayOfMaps(t *testing.T) {
|
||||
path := "[0].id"
|
||||
data := `[{"id": 1}, {"id": 2}]`
|
||||
|
||||
expectedOutput := "1"
|
||||
|
||||
output, _, err := Eval(path, []byte(data))
|
||||
if err != nil {
|
||||
t.Error("Didn't expect any error, but got", err)
|
||||
}
|
||||
if output != expectedOutput {
|
||||
t.Errorf("Expected output to be %v, but was %v", expectedOutput, output)
|
||||
}
|
||||
}
|
||||
|
||||
func TestEvalWithRootArrayOfMapsUsingInvalidArrayIndex(t *testing.T) {
|
||||
path := "[5].id"
|
||||
data := `[{"id": 1}, {"id": 2}]`
|
||||
|
||||
_, _, err := Eval(path, []byte(data))
|
||||
if err == nil {
|
||||
t.Error("Should've returned an error, but didn't")
|
||||
}
|
||||
}
|
||||
|
||||
func TestEvalWithLongWalkAndArray(t *testing.T) {
|
||||
path := "data.ids[0].id"
|
||||
data := `{"data": {"ids": [{"id": 1}, {"id": 2}, {"id": 3}]}}`
|
||||
|
||||
expectedOutput := "1"
|
||||
|
||||
output, _, err := Eval(path, []byte(data))
|
||||
if err != nil {
|
||||
t.Error("Didn't expect any error, but got", err)
|
||||
}
|
||||
if output != expectedOutput {
|
||||
t.Errorf("Expected output to be %v, but was %v", expectedOutput, output)
|
||||
}
|
||||
}
|
||||
|
||||
func TestEvalWithNestedArray(t *testing.T) {
|
||||
path := "[3][2]"
|
||||
data := `[[1, 2], [3, 4], [], [5, 6, 7]]`
|
||||
|
||||
expectedOutput := "7"
|
||||
|
||||
output, _, err := Eval(path, []byte(data))
|
||||
if err != nil {
|
||||
t.Error("Didn't expect any error, but got", err)
|
||||
}
|
||||
if output != expectedOutput {
|
||||
t.Errorf("Expected output to be %v, but was %v", expectedOutput, output)
|
||||
}
|
||||
}
|
||||
|
||||
func TestEvalWithMapOfNestedArray(t *testing.T) {
|
||||
path := "data[1][1]"
|
||||
data := `{"data": [["a", "b", "c"], ["d", "e", "f"]]}`
|
||||
|
||||
expectedOutput := "e"
|
||||
|
||||
output, _, err := Eval(path, []byte(data))
|
||||
if err != nil {
|
||||
t.Error("Didn't expect any error, but got", err)
|
||||
}
|
||||
if output != expectedOutput {
|
||||
t.Errorf("Expected output to be %v, but was %v", expectedOutput, output)
|
||||
for _, scenario := range scenarios {
|
||||
t.Run(scenario.Name, func(t *testing.T) {
|
||||
output, outputLength, err := Eval(scenario.Path, []byte(scenario.Data))
|
||||
if (err != nil) != scenario.ExpectedError {
|
||||
if scenario.ExpectedError {
|
||||
t.Errorf("Expected error, got '%v'", err)
|
||||
} else {
|
||||
t.Errorf("Expected no error, got '%v'", err)
|
||||
}
|
||||
}
|
||||
if outputLength != scenario.ExpectedOutputLength {
|
||||
t.Errorf("Expected output length to be %v, but was %v", scenario.ExpectedOutputLength, outputLength)
|
||||
}
|
||||
if output != scenario.ExpectedOutput {
|
||||
t.Errorf("Expected output to be %v, but was %v", scenario.ExpectedOutput, output)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
2
vendor/github.com/TwinProduction/gocache/Dockerfile
generated
vendored
2
vendor/github.com/TwinProduction/gocache/Dockerfile
generated
vendored
@@ -2,7 +2,7 @@
|
||||
FROM golang:alpine as builder
|
||||
WORKDIR /app
|
||||
ADD . ./
|
||||
RUN CGO_ENABLED=0 GOOS=linux go build -mod vendor -a -installsuffix cgo -o bin/gocache-server ./gocacheserver/main
|
||||
RUN CGO_ENABLED=0 GOOS=linux go build -mod vendor -a -installsuffix cgo -o bin/gocache-server cmd/server/main.go
|
||||
RUN apk --update add --no-cache ca-certificates
|
||||
|
||||
FROM scratch
|
||||
|
||||
2
vendor/github.com/TwinProduction/gocache/LICENSE.md
generated
vendored
2
vendor/github.com/TwinProduction/gocache/LICENSE.md
generated
vendored
@@ -1,6 +1,6 @@
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2020 TwinProduction
|
||||
Copyright (c) 2021 TwinProduction
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
|
||||
|
||||
|
||||
2
vendor/github.com/TwinProduction/gocache/Makefile
generated
vendored
2
vendor/github.com/TwinProduction/gocache/Makefile
generated
vendored
@@ -12,7 +12,7 @@ docker-run-max-memory-usage:
|
||||
docker run -p 6666:6379 -e AUTOSAVE=true -e MAX_CACHE_SIZE=0 -e MAX_MEMORY_USAGE=524288000 --name gocache-server -d gocache-server
|
||||
|
||||
run:
|
||||
PORT=6666 go run gocacheserver/main/server.go
|
||||
PORT=6666 go run cmd/server/main.go
|
||||
|
||||
start-redis:
|
||||
docker run -p 6379:6379 --name redis -d redis
|
||||
|
||||
216
vendor/github.com/TwinProduction/gocache/README.md
generated
vendored
216
vendor/github.com/TwinProduction/gocache/README.md
generated
vendored
@@ -5,6 +5,7 @@
|
||||
[](https://codecov.io/gh/TwinProduction/gocache)
|
||||
[](https://github.com/TwinProduction/gocache)
|
||||
[](https://pkg.go.dev/github.com/TwinProduction/gocache)
|
||||
[](https://github.com/TwinProduction)
|
||||
|
||||
gocache is an easy-to-use, high-performance, lightweight and thread-safe (goroutine-safe) in-memory key-value cache
|
||||
with support for LRU and FIFO eviction policies as well as expiration, bulk operations and even persistence to file.
|
||||
@@ -33,6 +34,8 @@ with support for LRU and FIFO eviction policies as well as expiration, bulk oper
|
||||
- [Summary](#summary)
|
||||
- [Results](#results)
|
||||
- [FAQ](#faq)
|
||||
- [How can I persist the data on application termination?](#how-can-i-persist-the-data-on-application-termination)
|
||||
- [How can I automatically save the cache to a file every 5 minutes?](#how-can-i-automatically-save-the-cache-to-a-file-every-5-minutes)
|
||||
- [Why does the memory usage not go down?](#why-does-the-memory-usage-not-go-down)
|
||||
|
||||
|
||||
@@ -58,6 +61,9 @@ It may also serve as a good reference to use in order to implement gocache in yo
|
||||
go get -u github.com/TwinProduction/gocache
|
||||
```
|
||||
|
||||
If you're interested in using gocache as a server rather than an embedded library, see [Server](#server)
|
||||
|
||||
|
||||
### Initializing the cache
|
||||
```go
|
||||
cache := gocache.NewCache().WithMaxSize(1000).WithEvictionPolicy(gocache.LeastRecentlyUsed)
|
||||
@@ -125,52 +131,52 @@ You can also delete multiple entries by using `cache.DeleteAll([]string{"key1",
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"time"
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"github.com/TwinProduction/gocache"
|
||||
)
|
||||
|
||||
func main() {
|
||||
cache := gocache.NewCache().WithEvictionPolicy(gocache.LeastRecentlyUsed).WithMaxSize(10000)
|
||||
cache.StartJanitor() // Passively manages expired entries
|
||||
cache := gocache.NewCache().WithEvictionPolicy(gocache.LeastRecentlyUsed).WithMaxSize(10000)
|
||||
cache.StartJanitor() // Passively manages expired entries
|
||||
|
||||
cache.Set("key", "value")
|
||||
cache.SetWithTTL("key-with-ttl", "value", 60*time.Minute)
|
||||
cache.SetAll(map[string]interface{}{"k1": "v1", "k2": "v2", "k3": "v3"})
|
||||
cache.Set("key", "value")
|
||||
cache.SetWithTTL("key-with-ttl", "value", 60*time.Minute)
|
||||
cache.SetAll(map[string]interface{}{"k1": "v1", "k2": "v2", "k3": "v3"})
|
||||
|
||||
value, exists := cache.Get("key")
|
||||
fmt.Printf("[Get] key=key; value=%s; exists=%v\n", value, exists)
|
||||
for key, value := range cache.GetByKeys([]string{"k1", "k2", "k3"}) {
|
||||
fmt.Printf("[GetByKeys] key=%s; value=%s\n", key, value)
|
||||
}
|
||||
for _, key := range cache.GetKeysByPattern("key*", 0) {
|
||||
fmt.Printf("[GetKeysByPattern] key=%s\n", key)
|
||||
}
|
||||
value, exists := cache.Get("key")
|
||||
fmt.Printf("[Get] key=key; value=%s; exists=%v\n", value, exists)
|
||||
for key, value := range cache.GetByKeys([]string{"k1", "k2", "k3"}) {
|
||||
fmt.Printf("[GetByKeys] key=%s; value=%s\n", key, value)
|
||||
}
|
||||
for _, key := range cache.GetKeysByPattern("key*", 0) {
|
||||
fmt.Printf("[GetKeysByPattern] key=%s\n", key)
|
||||
}
|
||||
|
||||
fmt.Println("Cache size before persisting cache to file:", cache.Count())
|
||||
err := cache.SaveToFile("cache.bak")
|
||||
if err != nil {
|
||||
panic(fmt.Sprintf("failed to persist cache to file: %s", err.Error()))
|
||||
}
|
||||
fmt.Println("Cache size before persisting cache to file:", cache.Count())
|
||||
err := cache.SaveToFile("cache.bak")
|
||||
if err != nil {
|
||||
panic(fmt.Sprintf("failed to persist cache to file: %s", err.Error()))
|
||||
}
|
||||
|
||||
cache.Expire("key", time.Hour)
|
||||
time.Sleep(500*time.Millisecond)
|
||||
timeUntilExpiration, _ := cache.TTL("key")
|
||||
fmt.Println("Number of minutes before 'key' expires:", int(timeUntilExpiration.Seconds()))
|
||||
cache.Expire("key", time.Hour)
|
||||
time.Sleep(500*time.Millisecond)
|
||||
timeUntilExpiration, _ := cache.TTL("key")
|
||||
fmt.Println("Number of minutes before 'key' expires:", int(timeUntilExpiration.Seconds()))
|
||||
|
||||
cache.Delete("key")
|
||||
cache.DeleteAll([]string{"k1", "k2", "k3"})
|
||||
cache.Delete("key")
|
||||
cache.DeleteAll([]string{"k1", "k2", "k3"})
|
||||
|
||||
fmt.Println("Cache size before restoring cache from file:", cache.Count())
|
||||
_, err = cache.ReadFromFile("cache.bak")
|
||||
if err != nil {
|
||||
panic(fmt.Sprintf("failed to restore cache from file: %s", err.Error()))
|
||||
}
|
||||
fmt.Println("Cache size before restoring cache from file:", cache.Count())
|
||||
_, err = cache.ReadFromFile("cache.bak")
|
||||
if err != nil {
|
||||
panic(fmt.Sprintf("failed to restore cache from file: %s", err.Error()))
|
||||
}
|
||||
|
||||
fmt.Println("Cache size after restoring cache from file:", cache.Count())
|
||||
cache.Clear()
|
||||
fmt.Println("Cache size after clearing the cache:", cache.Count())
|
||||
fmt.Println("Cache size after restoring cache from file:", cache.Count())
|
||||
cache.Clear()
|
||||
fmt.Println("Cache size after clearing the cache:", cache.Count())
|
||||
}
|
||||
```
|
||||
|
||||
@@ -215,8 +221,8 @@ While you can cache structs in memory out of the box, persisting structs to a fi
|
||||
|
||||
```go
|
||||
type YourCustomStruct struct {
|
||||
A string
|
||||
B int
|
||||
A string
|
||||
B int
|
||||
}
|
||||
|
||||
// ...
|
||||
@@ -251,6 +257,13 @@ every key that cannot be parsed are not populated into the cache by `ReadFromFil
|
||||
In other words, if you're falling back to a database or something similar when the cache doesn't have the key requested,
|
||||
you'll be fine.
|
||||
|
||||
Note that if you need to modify the type of a variable in a struct, you should change the name of that variable as well.
|
||||
For instance, if the struct has a `CreatedAt` variable with the type `time.Time` and that variable type is later
|
||||
modified to `uint64`, decoding the struct would fail, however, if you rename the variable to `CreatedAtUnixTimeInMs`,
|
||||
there won't be any decoding issues other than the loss of data for that field. You could also obviously handle the
|
||||
migration gracefully by keeping both variables, populating the `CreatedAtUnixTimeInMs` variable with the `CreatedAt`
|
||||
value and then removing the `CreatedAt` field.
|
||||
|
||||
|
||||
## Eviction
|
||||
|
||||
@@ -303,31 +316,37 @@ If you do not start the janitor, there will be no passive deletion of expired ke
|
||||
|
||||
|
||||
## Server
|
||||
For the sake of convenience, a ready-to-go cache server is available
|
||||
through the `gocacheserver` package.
|
||||
|
||||
The reason why the server is in a different package is because `gocache` limit its external dependencies to the strict
|
||||
minimum (e.g. boltdb for persistence), however, rather than re-inventing the wheel, the server implementation uses
|
||||
redcon, which is a very good Redis server framework for Go.
|
||||
|
||||
That way, those who desire to use gocache without the server will not add any extra dependencies
|
||||
as long as they don't import the `gocacheserver` package.
|
||||
For the sake of convenience, a ready-to-go cache server is available through the `server` package.
|
||||
|
||||
#### As an application
|
||||
```go
|
||||
package main
|
||||
|
||||
import (
|
||||
"github.com/TwinProduction/gocache"
|
||||
"github.com/TwinProduction/gocache/gocacheserver"
|
||||
"github.com/TwinProduction/gocache"
|
||||
gocacheserver "github.com/TwinProduction/gocache/server"
|
||||
)
|
||||
|
||||
func main() {
|
||||
cache := gocache.NewCache().WithEvictionPolicy(gocache.LeastRecentlyUsed).WithMaxSize(100000)
|
||||
server := gocacheserver.NewServer(cache).WithPort(6379)
|
||||
server.Start()
|
||||
cache := gocache.NewCache().WithEvictionPolicy(gocache.LeastRecentlyUsed).WithMaxSize(100000)
|
||||
server := gocacheserver.NewServer(cache).WithPort(6379)
|
||||
// This is a blocking function, therefore, you are expected to run this on a goroutine
|
||||
server.Start()
|
||||
}
|
||||
```
|
||||
|
||||
The reason why the server is in a different package is because `gocache` limit its external dependencies to the strict
|
||||
minimum (e.g. boltdb for persistence), however, rather than re-inventing the wheel, the server implementation uses
|
||||
redcon, which is a very good Redis server framework for Go.
|
||||
|
||||
That way, those who desire to use gocache without the server will not add any extra dependencies
|
||||
as long as they don't import the `server` package.
|
||||
|
||||
If you'd like to run it through the CLI:
|
||||
```
|
||||
go run cmd/server/main.go
|
||||
```
|
||||
|
||||
Any Redis client should be able to interact with the server, though only the following instructions are supported:
|
||||
- [X] GET
|
||||
- [X] SET
|
||||
@@ -350,14 +369,12 @@ Any Redis client should be able to interact with the server, though only the fol
|
||||
## Running the server with Docker
|
||||
[](https://cloud.docker.com/repository/docker/twinproduction/gocache-server)
|
||||
|
||||
To build it locally, refer to the Makefile's `docker-build` and `docker-run` steps.
|
||||
|
||||
Note that the server version of gocache is still under development.
|
||||
|
||||
```
|
||||
docker run --name gocache-server -p 6379:6379 twinproduction/gocache-server
|
||||
```
|
||||
|
||||
To build it locally, refer to the Makefile's `docker-build` and `docker-run` steps.
|
||||
|
||||
|
||||
## Performance
|
||||
|
||||
@@ -448,9 +465,96 @@ WithForceNilInterfaceOnNilPointerWithConcurrency/false-8
|
||||
|
||||
## FAQ
|
||||
|
||||
### How can I persist the data on application termination?
|
||||
|
||||
Because this library doesn't persist immediately after every write operations, persistence is instead expected to be
|
||||
done on a schedule, like for instance, every 10 minutes.
|
||||
|
||||
While this prevents you from losing all of your data, you may still lose some data if the application stopped 9 minutes
|
||||
after the previous "auto save".
|
||||
|
||||
To increase your odds of not losing any data, you can use Go's `signal` package, more specifically its `Notify` function
|
||||
which allows listening for termination signals like SIGTERM and SIGINT. Once a termination signal is caught, you can
|
||||
add the necessary logic for a graceful shutdown.
|
||||
|
||||
In the following example, the code that would usually be present in the `main` function is moved to a different function
|
||||
named `Start` which is launched on a different goroutine so that listening for a termination signals is what blocks the
|
||||
main goroutine instead:
|
||||
|
||||
```go
|
||||
package main
|
||||
|
||||
import (
|
||||
"log"
|
||||
"os"
|
||||
"os/signal"
|
||||
"syscall"
|
||||
|
||||
"github.com/TwinProduction/gocache"
|
||||
)
|
||||
|
||||
const CacheFile = "gocache.data"
|
||||
|
||||
var cache = gocache.NewCache()
|
||||
|
||||
func main() {
|
||||
// Load persisted data from file
|
||||
cache.ReadFromFile(CacheFile)
|
||||
// Start everything else on another goroutine to prevent blocking the main goroutine
|
||||
go Start()
|
||||
// Wait for termination signal
|
||||
sig := make(chan os.Signal, 1)
|
||||
done := make(chan bool, 1)
|
||||
signal.Notify(sig, os.Interrupt, syscall.SIGTERM)
|
||||
go func() {
|
||||
<-sig
|
||||
log.Println("Received termination signal, attempting to gracefully shut down")
|
||||
err := cache.SaveToFile(CacheFile)
|
||||
if err != nil {
|
||||
log.Println("Failed to save storage provider:", err.Error())
|
||||
}
|
||||
done <- true
|
||||
}()
|
||||
<-done
|
||||
log.Println("Shutting down")
|
||||
}
|
||||
```
|
||||
|
||||
Note that this won't protect you from a SIGKILL, as this signal cannot be caught.
|
||||
|
||||
|
||||
### How can I automatically save the cache to a file every 5 minutes?
|
||||
|
||||
Beside using the suggestion above, automatically persisting the cache on an interval will protect your application from
|
||||
sudden terminations triggered by signals that cannot be caught, such as the force kill signal received by an application
|
||||
being OOMKilled.
|
||||
|
||||
The simplest implementation could be something like this:
|
||||
```go
|
||||
const CacheFile = "gocache.data"
|
||||
|
||||
func main() {
|
||||
cache := gocache.NewCache()
|
||||
cache.ReadFromFile(CacheFile)
|
||||
go autoSave(10*time.Minute)
|
||||
// ...
|
||||
}
|
||||
|
||||
func autoSave(interval time.Duration) {
|
||||
for {
|
||||
err := cache.SaveToFile(CacheFile)
|
||||
if err != nil {
|
||||
log.Println("Failed to persist cache to file:", err.Error())
|
||||
}
|
||||
time.Sleep(interval)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
### Why does the memory usage not go down?
|
||||
|
||||
> **NOTE**: As of Go 1.16, this will no longer apply. See [golang/go#42330](https://github.com/golang/go/issues/42330)
|
||||
> **NOTE**: As of Go 1.16, this no longer applies. See [golang/go#42330](https://github.com/golang/go/issues/42330)
|
||||
|
||||
By default, Go uses `MADV_FREE` if the kernel supports it to release memory, which is significantly more efficient
|
||||
than using `MADV_DONTNEED`. Unfortunately, this means that RSS doesn't go down unless the OS actually needs the
|
||||
@@ -463,7 +567,7 @@ notice the memory usage lowering.
|
||||
[reference](https://github.com/golang/go/issues/33376#issuecomment-666455792)
|
||||
|
||||
You can reproduce this by following the steps below:
|
||||
- Start gocacheserver
|
||||
- Start the server
|
||||
- Note the memory usage
|
||||
- Create 500k keys
|
||||
- Note the memory usage
|
||||
|
||||
2
vendor/github.com/TwinProduction/gocache/go.mod
generated
vendored
2
vendor/github.com/TwinProduction/gocache/go.mod
generated
vendored
@@ -1,6 +1,6 @@
|
||||
module github.com/TwinProduction/gocache
|
||||
|
||||
go 1.15
|
||||
go 1.16
|
||||
|
||||
require (
|
||||
github.com/go-redis/redis v6.15.9+incompatible
|
||||
|
||||
4
vendor/github.com/TwinProduction/gocache/go.sum
generated
vendored
4
vendor/github.com/TwinProduction/gocache/go.sum
generated
vendored
@@ -9,11 +9,9 @@ github.com/golang/protobuf v1.4.0-rc.1.0.20200221234624-67d41d38c208/go.mod h1:x
|
||||
github.com/golang/protobuf v1.4.0-rc.2/go.mod h1:LlEzMj4AhA7rCAGe4KMBDvJI+AwstrUpVNzEA03Pprs=
|
||||
github.com/golang/protobuf v1.4.0-rc.4.0.20200313231945-b860323f09d0/go.mod h1:WU3c8KckQ9AFe+yFwt9sWVRKCVIyN9cPHBJSNnbL67w=
|
||||
github.com/golang/protobuf v1.4.0/go.mod h1:jodUvKwWbYaEsadDk5Fwe5c77LiNKVO9IDvqG2KuDX0=
|
||||
github.com/golang/protobuf v1.4.2 h1:+Z5KGCizgyZCbGh1KZqA0fcLLkwbsjIzS4aV2v7wJX0=
|
||||
github.com/golang/protobuf v1.4.2/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI=
|
||||
github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
|
||||
github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
|
||||
github.com/google/go-cmp v0.4.0 h1:xsAVV57WRhGj6kEIi8ReJzQlHHqcBYCElAvkovg3B/4=
|
||||
github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
|
||||
github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU=
|
||||
github.com/nxadm/tail v1.4.4 h1:DQuhQpB1tVlglWS2hLQ5OV6B5r8aGxSrPc5Qo6uTN78=
|
||||
@@ -55,9 +53,7 @@ google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ
|
||||
google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM=
|
||||
google.golang.org/protobuf v1.20.1-0.20200309200217-e05f789c0967/go.mod h1:A+miEFZTKqfCUM6K7xSMQL9OKL/b6hQv+e19PK+JZNE=
|
||||
google.golang.org/protobuf v1.21.0/go.mod h1:47Nbq4nVaFHyn7ilMalzfO3qCViNmqZ2kzikPIcrTAo=
|
||||
google.golang.org/protobuf v1.23.0 h1:4MY060fB1DLGMB/7MBTLnwQUY6+F09GEiz6SsrNqyzM=
|
||||
google.golang.org/protobuf v1.23.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||
gopkg.in/fsnotify.v1 v1.4.7/go.mod h1:Tz8NjZHkW78fSQdbUxIjBTcgA1z1m8ZHf0WmKUhAMys=
|
||||
gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 h1:uRGJdciOHaEIrze2W8Q3AKkepLTh2hOroT7a+7czfdQ=
|
||||
|
||||
20
vendor/github.com/TwinProduction/gocache/gocache.go
generated
vendored
20
vendor/github.com/TwinProduction/gocache/gocache.go
generated
vendored
@@ -239,12 +239,12 @@ func (cache *Cache) SetWithTTL(key string, value interface{}, ttl time.Duration)
|
||||
Key: key,
|
||||
Value: value,
|
||||
RelevantTimestamp: time.Now(),
|
||||
previous: cache.head,
|
||||
next: cache.head,
|
||||
}
|
||||
if cache.head == nil {
|
||||
cache.tail = entry
|
||||
} else {
|
||||
cache.head.next = entry
|
||||
cache.head.previous = entry
|
||||
}
|
||||
cache.head = entry
|
||||
cache.entries[key] = entry
|
||||
@@ -260,7 +260,7 @@ func (cache *Cache) SetWithTTL(key string, value interface{}, ttl time.Duration)
|
||||
return
|
||||
}
|
||||
if cache.maxMemoryUsage != NoMaxMemoryUsage {
|
||||
// Substract the old entry from the cache's memoryUsage
|
||||
// Subtract the old entry from the cache's memoryUsage
|
||||
cache.memoryUsage -= entry.SizeInBytes()
|
||||
}
|
||||
// Update existing entry's value
|
||||
@@ -278,8 +278,8 @@ func (cache *Cache) SetWithTTL(key string, value interface{}, ttl time.Duration)
|
||||
} else {
|
||||
entry.Expiration = NoExpiration
|
||||
}
|
||||
// If the cache doesn't have a maxSize/maxMemoryUsage, then there's no point checking if we need to evict
|
||||
// an entry, so we'll just return now
|
||||
// If the cache doesn't have a maxSize/maxMemoryUsage, then there's no point
|
||||
// checking if we need to evict an entry, so we'll just return now
|
||||
if cache.maxSize == NoMaxSize && cache.maxMemoryUsage == NoMaxMemoryUsage {
|
||||
cache.mutex.Unlock()
|
||||
return
|
||||
@@ -519,10 +519,10 @@ func (cache *Cache) moveExistingEntryToHead(entry *Entry) {
|
||||
cache.removeExistingEntryReferences(entry)
|
||||
}
|
||||
if entry != cache.head {
|
||||
entry.previous = cache.head
|
||||
entry.next = nil
|
||||
entry.next = cache.head
|
||||
entry.previous = nil
|
||||
if cache.head != nil {
|
||||
cache.head.next = entry
|
||||
cache.head.previous = entry
|
||||
}
|
||||
cache.head = entry
|
||||
}
|
||||
@@ -536,9 +536,9 @@ func (cache *Cache) removeExistingEntryReferences(entry *Entry) {
|
||||
cache.tail = nil
|
||||
cache.head = nil
|
||||
} else if cache.tail == entry {
|
||||
cache.tail = cache.tail.next
|
||||
cache.tail = cache.tail.previous
|
||||
} else if cache.head == entry {
|
||||
cache.head = cache.head.previous
|
||||
cache.head = cache.head.next
|
||||
}
|
||||
if entry.previous != nil {
|
||||
entry.previous.next = entry.next
|
||||
|
||||
23
vendor/github.com/TwinProduction/gocache/janitor.go
generated
vendored
23
vendor/github.com/TwinProduction/gocache/janitor.go
generated
vendored
@@ -33,7 +33,7 @@ func (cache *Cache) StartJanitor() error {
|
||||
}
|
||||
cache.stopJanitor = make(chan bool)
|
||||
go func() {
|
||||
// rather than starting from the tail on every run, we can try to start from the last next entry
|
||||
// rather than starting from the tail on every run, we can try to start from the last traversed entry
|
||||
var lastTraversedNode *Entry
|
||||
totalNumberOfExpiredKeysInPreviousRunFromTailToHead := 0
|
||||
backOff := JanitorMinShiftBackOff
|
||||
@@ -62,13 +62,14 @@ func (cache *Cache) StartJanitor() error {
|
||||
totalNumberOfExpiredKeysInPreviousRunFromTailToHead = 0
|
||||
}
|
||||
for current != nil {
|
||||
var next *Entry
|
||||
// since we're walking from the tail to the head, we get the previous reference
|
||||
var previous *Entry
|
||||
steps++
|
||||
if current.Expired() {
|
||||
expiredEntriesFound++
|
||||
// Because delete will remove the next reference from the entry, we need to store the
|
||||
// next reference before we delete it
|
||||
next = current.next
|
||||
// Because delete will remove the previous reference from the entry, we need to store the
|
||||
// previous reference before we delete it
|
||||
previous = current.previous
|
||||
cache.delete(current.Key)
|
||||
cache.stats.ExpiredKeys++
|
||||
}
|
||||
@@ -76,11 +77,11 @@ func (cache *Cache) StartJanitor() error {
|
||||
lastTraversedNode = nil
|
||||
break
|
||||
}
|
||||
// Travel to the current node's next node only if no specific next node has been specified
|
||||
if next != nil {
|
||||
current = next
|
||||
// Travel to the current node's previous node only if no specific previous node has been specified
|
||||
if previous != nil {
|
||||
current = previous
|
||||
} else {
|
||||
current = current.next
|
||||
current = current.previous
|
||||
}
|
||||
lastTraversedNode = current
|
||||
if steps == JanitorMaxIterationsPerShift || expiredEntriesFound >= JanitorShiftTarget {
|
||||
@@ -131,8 +132,8 @@ func (cache *Cache) StartJanitor() error {
|
||||
func (cache *Cache) StopJanitor() {
|
||||
if cache.stopJanitor != nil {
|
||||
// Tell the janitor to stop, and then wait for the janitor to reply on the same channel that it's stopping
|
||||
// This may seem a bit odd, but this allows us to avoid a data race condition in which setting cache.stopJanitor
|
||||
// to nil
|
||||
// This may seem a bit odd, but this allows us to avoid a data race condition when trying to set
|
||||
// cache.stopJanitor to nil
|
||||
cache.stopJanitor <- true
|
||||
<-cache.stopJanitor
|
||||
cache.stopJanitor = nil
|
||||
|
||||
10
vendor/github.com/TwinProduction/gocache/persistence.go
generated
vendored
10
vendor/github.com/TwinProduction/gocache/persistence.go
generated
vendored
@@ -85,6 +85,12 @@ func (cache *Cache) ReadFromFile(path string) (int, error) {
|
||||
if err != nil {
|
||||
// Failed to decode the value, so we'll skip it.
|
||||
// This is likely due to the fact that the custom struct wasn't registered using gob.Register(...)
|
||||
//
|
||||
// Could also be due to a breaking change in a struct's variable. For instance, if the struct has
|
||||
// a variable with a type map[string]string and that variable is modified to map[string]int,
|
||||
// decoding the struct would fail. This can be avoided by using a different variable name every
|
||||
// time you must change the type of a variable within a struct.
|
||||
//
|
||||
// See [Persistence - Limitations](https://github.com/TwinProduction/gocache#limitations)
|
||||
return err
|
||||
}
|
||||
@@ -114,8 +120,8 @@ func (cache *Cache) ReadFromFile(path string) (int, error) {
|
||||
cache.tail = current
|
||||
cache.head = current
|
||||
} else {
|
||||
previous.next = current
|
||||
current.previous = previous
|
||||
previous.previous = current
|
||||
current.next = previous
|
||||
cache.head = current
|
||||
}
|
||||
previous = entries[i]
|
||||
|
||||
1
vendor/github.com/TwinProduction/gocache/policy.go
generated
vendored
1
vendor/github.com/TwinProduction/gocache/policy.go
generated
vendored
@@ -1,5 +1,6 @@
|
||||
package gocache
|
||||
|
||||
// EvictionPolicy is what dictates how evictions are handled
|
||||
type EvictionPolicy string
|
||||
|
||||
var (
|
||||
|
||||
2
vendor/modules.txt
vendored
2
vendor/modules.txt
vendored
@@ -1,7 +1,7 @@
|
||||
# cloud.google.com/go v0.74.0
|
||||
## explicit
|
||||
cloud.google.com/go/compute/metadata
|
||||
# github.com/TwinProduction/gocache v1.2.1
|
||||
# github.com/TwinProduction/gocache v1.2.3
|
||||
## explicit
|
||||
github.com/TwinProduction/gocache
|
||||
# github.com/TwinProduction/health v1.0.0
|
||||
|
||||
@@ -1,24 +1,20 @@
|
||||
<template>
|
||||
<div id="settings">
|
||||
<div class="flex">
|
||||
<div class="flex bg-gray-200 border-gray-300 rounded border shadow dark:text-gray-200 dark:bg-gray-800 dark:border-gray-500">
|
||||
<div class="text-xs text-gray-600 rounded-xl py-1 px-2 dark:text-gray-200">
|
||||
↻
|
||||
</div>
|
||||
<select class="text-center text-gray-500 text-xs dark:text-gray-200 dark:bg-gray-800 border-r border-l border-gray-300 dark:border-gray-500" id="refresh-rate" ref="refreshInterval" @change="handleChangeRefreshInterval">
|
||||
<option value="10" :selected="refreshInterval === 10">10s</option>
|
||||
<option value="30" :selected="refreshInterval === 30">30s</option>
|
||||
<option value="60" :selected="refreshInterval === 60">1m</option>
|
||||
<option value="120" :selected="refreshInterval === 120">2m</option>
|
||||
<option value="300" :selected="refreshInterval === 300">5m</option>
|
||||
<option value="600" :selected="refreshInterval === 600">10m</option>
|
||||
</select>
|
||||
<button @click="toggleDarkMode" class="text-xs p-1">
|
||||
<slot v-if="darkMode">☀</slot>
|
||||
<slot v-else>🌙</slot>
|
||||
</button>
|
||||
</div>
|
||||
<div id="settings" class="flex bg-gray-200 border-gray-300 rounded border shadow dark:text-gray-200 dark:bg-gray-800 dark:border-gray-500">
|
||||
<div class="text-xs text-gray-600 rounded-xl py-1 px-2 dark:text-gray-200">
|
||||
↻
|
||||
</div>
|
||||
<select class="text-center text-gray-500 text-xs dark:text-gray-200 dark:bg-gray-800 border-r border-l border-gray-300 dark:border-gray-500" id="refresh-rate" ref="refreshInterval" @change="handleChangeRefreshInterval">
|
||||
<option value="10" :selected="refreshInterval === 10">10s</option>
|
||||
<option value="30" :selected="refreshInterval === 30">30s</option>
|
||||
<option value="60" :selected="refreshInterval === 60">1m</option>
|
||||
<option value="120" :selected="refreshInterval === 120">2m</option>
|
||||
<option value="300" :selected="refreshInterval === 300">5m</option>
|
||||
<option value="600" :selected="refreshInterval === 600">10m</option>
|
||||
</select>
|
||||
<button @click="toggleDarkMode" class="text-xs p-1">
|
||||
<slot v-if="darkMode">☀</slot>
|
||||
<slot v-else>🌙</slot>
|
||||
</button>
|
||||
</div>
|
||||
</template>
|
||||
|
||||
@@ -83,12 +79,11 @@ export default {
|
||||
</script>
|
||||
|
||||
|
||||
<style scoped>
|
||||
<style>
|
||||
#settings {
|
||||
position: fixed;
|
||||
left: 5px;
|
||||
bottom: 5px;
|
||||
padding: 5px;
|
||||
left: 10px;
|
||||
bottom: 10px;
|
||||
}
|
||||
|
||||
#settings select:focus {
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -1 +1 @@
|
||||
<!doctype html><html lang="en"><head><meta charset="utf-8"><meta http-equiv="X-UA-Compatible" content="IE=edge"><meta name="viewport" content="width=device-width,initial-scale=1"><link rel="icon" href="/favicon.ico"><title>Health Dashboard | Gatus</title><script defer="defer" src="/js/chunk-vendors.js" type="module"></script><script defer="defer" src="/js/app.js" type="module"></script><link href="/css/app.css" rel="stylesheet"><script defer="defer" src="/js/chunk-vendors-legacy.js" nomodule></script><script defer="defer" src="/js/app-legacy.js" nomodule></script></head><body class="dark:bg-gray-900"><noscript><strong>Enable JavaScript to view this page.</strong></noscript><div id="app"></div></body></html>
|
||||
<!doctype html><html lang="en"><head><meta charset="utf-8"/><title>Health Dashboard | Gatus</title><meta http-equiv="X-UA-Compatible" content="IE=edge"><meta name="viewport" content="width=device-width,initial-scale=1"><link rel="icon" href="/favicon.ico"><script defer="defer" src="/js/chunk-vendors.js" type="module"></script><script defer="defer" src="/js/app.js" type="module"></script><link href="/css/app.css" rel="stylesheet"><script defer="defer" src="/js/chunk-vendors-legacy.js" nomodule></script><script defer="defer" src="/js/app-legacy.js" nomodule></script></head><body class="dark:bg-gray-900"><noscript><strong>Enable JavaScript to view this page.</strong></noscript><div id="app"></div></body></html>
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
Reference in New Issue
Block a user