### What
This patches two very subtle bugs in the proxy repository that cause it
to never actually stop polling the db in the background
## Details - Issue 1
We've recently started to get the following output when running `yarn
test`:
` Attempted to log "Error: Unable to acquire a connection
at Object.queryBuilder
(/home/simon/dev/unleash/node_modules/knex/lib/knex-builder/make-knex.js:111:26)`
This seems to occur for every test suite after running the proxy tests
and the full stack trace doesn't point to anything related to the
running tests that produce this output. Running a `git bisect` points to
this commit:
6e44a65c58
being the culprit but I believe that this may have surfaced the bug
rather than causing it.
Layering in a few console logs and running Unleash, seems to point to
the proxy repository setting up data polling but never actually
terminating it when `stop` was called, which is inline with the output
here - effectively the tests were continuing to run the polling in the
background after the suite had exited and jest freaks out that an async
task is running when it shouldn't be. This is easy to reproduce once the
console logs are in place in the `dataPolling` function, by running
Unleash - creating and deleting a front end token never terminates the
poll cycle.
I believe the cause here is some subtlety around using async functions
with timers - stop was being called, which results in the timer being
cleared but a scheduled async call was already on the stack, causing the
recursive call to resolve after stop, resurrecting the timer and
reinitializing the poll cycle.
I've moved the terminating code into the async callback. Which seems to
solve the problem here.
## Details - Issue 2
Related to the first issue, when the proxy service stops the underlying
Unleash Client, it never actually calls destroy on the client, it only
removes it from its internal map. That in turn means that the Client
never calls stop on the injected repository, it only removes it from
memory. However, the scheduled task is `async` and `unref`, meaning it
continues to spin in the background until every other process also
exits. This is patched by simply calling destroy on the client when
cleaning up
## The Ugly
This is really hard to test effectively, mostly because this is an issue
caused by internals within NodeJS and async. I've added a test that
reads the output from the debug log (and also placed a debug log in the
termination code). This also requires the test code to wait until the
async task completes. This is horribly fragile so if someone has a
better idea on how to prove this I would be a very happy human.
The second ugly part is that this is a subtle issue in complex code that
really, really needs to work correctly. I'm nervous about making changes
here without lots of eyes on this
## About the changes
client-metrics-schema is less strict than proxy-metrics-schema because
the former allows empty `instanceId` and also supports dates as
timestamps as well as date-formatted strings.
Using the same schema makes sense to reduce maintainability costs and
it's less error-prone if we need to modify the schema because underlying
the schema they both use the same code.
The reasoning is that proxy metrics should align with our client
metrics. Alternatively, we have new endpoints for edge metrics that will
aggregate and bucket by client.
## Discussion points
Will we ever want to evolve proxy-metrics differently than
client-metrics? I'm under the assumption that the answer is no
### What
Change /edge/metrics endpoint to accept list of ClientMetricsEnv
### Rationale
We originally made the assumption that we probably didn't need to keep
splitting from a map of features into ClientMetricsEnv for bulk, instead
the bulk poster could post ClientMetricsEnv directly. However, Unleash
still expected the old client metrics format with a dictionary of
featurename -> metricsForFeature. This PR changes that to now accept the
list of ClientMetricsEnv (preprocessed data from downstream) instead of
expecting metrics to be in the old single application metric format.
## About the changes
Currently, we need to remember of using the email or else the username
of a user when storing into EventStore, because we don't have
[strictNullChecks](https://www.typescriptlang.org/tsconfig#strictNullChecks),
it's error-prone. Fix for a production issue: #3072
This reuses an existing function that also deals with undefined
## About the changes
Promoted experimental networkView flag into a configuration that relies
on prometheusApi being configured.
Also, a follow-up on https://github.com/Unleash/unleash/pull/3054 moving
this code to enterprise because it doesn't make sense to maintain this
code in OSS where it's not being used.
## About the changes
Implementation of bulk metrics and registration endpoint. This will be
used by edge nodes to send all collected information.
Types around metrics were improved and `IClientApp.bucket` with type
`any` is no longer needed
---------
Co-authored-by: sighphyre <liquidwicked64@gmail.com>
This PR changes the behavior of checking the incoming token on the
`/api/frontend` path. Instead of using assert resulting in a 500 error
we are throwing an error that is caught by the default controller and
emitted back to the user as JSON.
This should be the correct behaviour, since the endpoint can not give
you any meaningful data without the environment that the API token
holds.
## About the changes
Spotted some issues in logs:
```json
{
"level":"warn",
"message":"Failed to store \"feature-environment-variants-updated\" event: error: insert into \"events\" (\"created_by\", \"data\", \"environment\", \"feature_name\", \"pre_data\", \"project\", \"tags\", \"type\") values (DEFAULT, $1, $2, $3, $4, $5, $6, $7) returning \"id\", \"type\", \"created_by\", \"created_at\", \"data\", \"pre_data\", \"tags\", \"feature_name\", \"project\", \"environment\" - null value in column \"created_by\" violates not-null constraint",
"name":"lib/db/event-store.ts"
}
```
In all other events we're doing the following:
b7fdcd36c0/src/lib/services/segment-service.ts (L80)
So this is just mimicking that to quickly release a patch, but I'll look
into a safer (type-checked) solution so this problem does not happen
again
This PR fixes two issues with events today:
1. Feature toggles "Event log" must include all events, regardless of
the project. This is important as feature toggles may move between
2. Add/remove tags on a feature toggle events should include project id
in order to show up in the project specific event log.
Batch Metrics as a capability developed to support the frontend API to
handle more metrics from SDKs without overloading the DB to much. It has
been running in Unleash Cloud for months and has proven to work quite
nice.
This PR simply removes the flag to make the capability GA, also for
self-hosted users.
This PR takes the project status API a step further by adding the
capability of providing a date to control the selection. We are
currently making calculations based on a gliding 30 day window, updated
once a day. The initial database structure and method for updating the
UI is outlined in this PR.
