https://linear.app/unleash/issue/2-3696/report-unknown-flags-when-sent-to-the-bulk-metrics-endpoint
Unifies metrics sifting logic across both metrics endpoints:
- `/metrics`
- `/metrics/bulk`
This PR improves consistency between the `/metrics` and `/metrics/bulk`
endpoints by introducing a shared `siftMetrics` method, now used within
`registerBulkMetrics`. Both endpoints already call this method at the
end of their respective logic flows, ensuring that metrics are sifted in
the same way regardless of the path taken.
While the primary goal was to enable reporting of unknown flags via the
`/metrics/bulk` endpoint, this change also improves bulk processing by
consistently dropping invalid or unknown flags before insertion, just
like in the regular `/metrics` endpoint.
Fixes a bug where `registerInstance` and
`register{Frontend|Backend}Client` would overwrite each other's data in
the instance service, leading to the bulk update being made with partial
data, often missing SDK version. There's a different issue in the actual
store that causes sdk version and type to be overwritten when it's
updated (because we don't use `setLastSeen` anymore), but I'll handle
that in a different PR.
This PR adds tests for the changes I've made. Additionally, I've made
these semi-related bonus changes:
- In registerInstance, don't expect a partial `IClientApp`. We used to
validate that it was actual a metrics object instead. Instead, update
the signature to expect the actual properties we need from the cilent
metrics schema and set a default for instanceId the way Joi did.
- In `metrics.ts`, use the `ClientMetricsSchema` type in the function
signature, so that the request body is correctly typed in the function
(instead of being `any`).
- Delete two unused properties from the`createApplicationSchema`. They
would get ignored and were never used as far as I can tell. (`appName`
is taken from the URL, and applications don't store `sdkVersion`
information).
- Add `sdkVersion` to `IClientApp` because it's used in instance
service.
I've been very confused about all the weird type shenanigans we do in
the instance service (expecting `IClientApp`, then validating with a
different Joi schema etc). I think this makes it a little bit better and
updates the bits I'm touching, but I'm happy to take input if you
disagree.
https://linear.app/unleash/issue/2-3695/allow-empty-flag-names-to-be-reported-in-bulk-metrics
Accepts metrics with empty flag names in the `/api/client/metrics/bulk`
endpoint.
When testing unknown flags through Edge, which uses the `/bulk`
endpoint, we noticed that there's a slight difference in validation
behavior compared to the regular metrics endpoint. While the regular
endpoint allows empty flag names, this one does not.
We can argue that we don't care about empty flag names in the first
place, which is true, but this inconsistency between the metric
endpoints can be confusing, and it also means that a single empty flag
name evaluation would break metrics being reported for that entire Edge
instance, for example.
This way we still accept it, just like we currently do if we point to
Unleash directly instead of going through Edge.
**Note**: We noticed that, due to the slightly different logic branch,
the bulk metrics endpoint does not report unknown flags. We'll take a
look at this at a later point.
This is primarily to facilitate reading and processing these events in
the payg cloud section of Unleash. We only emit these in one place, so I
added the types in there.
Accepts the new impact metrics into the singleton registry and then does
nothing with them. If the relevant flag is off, the metrics are stripped
from the existing metrics data format and dropped on the floor
**BREAKING CHANGE**: DEFAULT_ENV changed from `default` (should not be
used anymore) to `development`
## About the changes
- Only delete default env if the install is fresh new.
- Consider development the new default. The main consequence of this
change is that the default is no longer considered `type=production`
environment but also for frontend tokens due to this assumption:
724c4b78a2/src/lib/schema/api-token-schema.test.ts (L54-L59)
(I believe this is mostly due to the [support for admin
tokens](https://github.com/Unleash/unleash/pull/10080#discussion_r2126871567))
- `feature_toggle_update_total` metric reports `n/a` in environment and
environment type as it's not environment specific
Now we can receive custom metrics, return those for UI and have extra
prometheus endpoint for it.
---------
Co-authored-by: Christopher Kolstad <chriswk@getunleash.io>
Vitest Pros:
* Automated failing test comments on github PRs
* A nice local UI with incremental testing when changing files (`yarn
test:ui`)
* Also nicely supported in all major IDEs, click to run test works (so
we won't miss what we had with jest).
* Works well with ESM
Vitest Cons:
* The ESBuild transformer vitest uses takes a little longer to transform
than our current SWC/jest setup, however, it is possible to setup SWC as
the transformer for vitest as well (though it only does one transform,
so we're paying ~7-10 seconds instead of ~ 2-3 seconds in transform
phase).
* Exposes how slow our tests are (tongue in cheek here)
https://linear.app/unleash/issue/2-3564/remove-filterexistingflagnames-feature-flag
We're removing the `filterExistingFlagNames` feature flag since we've
decided we want this to be the default behavior.
We don't need to rush to merge it, just in case we need to disable this
for any reason. However it should also be pretty easy to just revert if
needed.
Changes in tests are a bit tricky since they assumed the previous
behavior where we always registered metrics, even for non existing flag
names. `cachedFeatureNames` is also memoized with a TTL of 10s, so the
easiest way to overcome this was to override `cachedFeatureNames` to
return what we expected. As long as they return the same flag names that
we expect, we're able to register their metrics.
Let me know if you can think of a better approach.
We're migrating to ESM, which will allow us to import the latest
versions of our dependencies.
Co-Authored-By: Christopher Kolstad <chriswk@getunleash.io>
https://linear.app/unleash/issue/2-3406/hold-unknown-flags-in-memory-and-show-them-in-the-ui-somehow
This PR introduces a suggestion for a “unknown flags” feature.
When clients report metrics for flags that don’t exist in Unleash (e.g.
due to typos), we now track a limited set of these unknown flag names
along with the appnames that reported them. The goal is to help users
identify and clean up incorrect flag usage across their apps.
We store up to 10 unknown flag + appName combinations, keeping only the
most recent reports. Data is collected in-memory and flushed
periodically to the DB, with deduplication and merging to ensure we
don’t exceed the cap even across pods.
We were especially careful to make this implementation defensive, as
unknown flags could be reported in very high volumes. Writes are
batched, deduplicated, and hard-capped to avoid DB pressure.
No UI has been added yet — this is backend-only for now and intended as
a step toward better visibility into client misconfigurations.
I would suggest starting with a simple banner that opens a dialog
showing the list of unknown flags and which apps reported them.
<img width="497" alt="image"
src="https://github.com/user-attachments/assets/b7348e0d-0163-4be4-a7f8-c072e8464331"
/>
This test was flaky once an hour because subminutes 3 made it fall into
the wrong bucket when tests were run exactly or minutes after the our
had passed.
Also, the databases created were created with the system clock. I
altered it to be explicitly UTC.
As part of preparation for ESM and node/TSC updates, this PR will make
Unleash build with strictNullChecks set to true, since that's what's in
our tsconfig file. Hence, this PR also removes the `--strictNullChecks
false` flag in our compile tasks in package.json.
TL;DR - Clean up your code rather than turning off compiler security
features :)
