This PR adds updates the potentially stale status change events whenever
the potentially stale update function is run.
No events are emitted yet. While the emission is only a few lines of
code, I'd like to do that in a separate PR so that we can give it the
attention it deserves in the form of tests, etc.
This PR also moves the potentially stale update functionality from the
`update` method to only being done in the
`updatePotentiallyStaleFeatures` method. This keeps all functionality
related to marking `potentiallyStale` in one place.
The emission implementation was removed in
4fb7cbde03
## The update queries
While it would be possible to do the state updates in a single query
instead of three separate ones, wrangling this into knex proved to be
troublesome (and would also probably be harder to understand and reason
about). The current solution uses three smaller queries (one select, two
updates), as Jaanus suggested in a private slack thread.
This PR lays most of the groundwork required for emitting events when
features are marked as potentially stale by Unleash. It does **not**
emit any events just yet. The summary is:
- periodically look for features that are potentially stale and mark
them (set to run every 10 seconds for now; can be changed)
- when features are updated, if the update data contains changes to the
feature's type or createdAt date, also update the potentially stale
status.
It is currently about 220 lines of tests and about 100 lines of
application code (primarily db migration and two new methods on the
IFeatureToggleStore interface).
The reason I wanted to put this into a single PR (instead of just the db
migration, then just the potentially stale marking, then the update
logic) is:
If users get the db migration first, but not the rest of the update
logic until the events are fired, then they could get a bunch of new
events for features that should have been marked as potentially stale
several days/weeks/months ago. That seemed undesirable to me, so I
decided to bunch those changes together. Of course, I'd be happy to
break it into smaller parts.
## Rules
A toggle will be marked as potentially stale iff:
- it is not already stale
- its createdAt date is older than its feature type's expected lifetime
would dictate
## Migration
The migration adds a new `potentially_stale` column to the features
table and sets this to true for any toggles that have exceeded their
expected lifetime and that have not already been marked as `stale`.
## Discussion
### The `currentTime` parameter of `markPotentiallyStaleFeatures`
The `markPotentiallyStaleFetaures` method takes an optional
`currentTime` parameter. This was added to make it easier to test (so
you can test "into the future"), but it's not used in the application.
We can rewrite the tests to instead update feature toggles manually, but
that wouldn't test the actual marking method. Happy to discuss.
This PR fixes an issue where events generated during a db transaction
would get published before the transaction was complete. This caused
errors in some of our services that expected the data to be stored
before the transaction had been commited. Refer to [linear issue
1-1049](https://linear.app/unleash/issue/1-1049/event-emitter-should-emit-events-after-db-transaction-is-commited-not)
for more info.
Fixes 1-1049.
## Changes
The most important change here is that the `eventStore` no longer emits
events when they happen (because that can be in the middle of a
transaction). Instead, events are stored with a new `announced` column.
The new event announcer service runs on a schedule (every second) and
publishes any new events that have not been published.
Parts of the code have largely been lifted from the
`client-application-store`, which uses a similar logic.
I have kept the emitting of the event within the event store because a
lot of other services listen to events from this store, so removing that
would require a large rewrite. It's something we could look into down
the line, but it seems like too much of a change to do right now.
## Discussion
### Terminology:
Published vs announced? We should settle on one or the other. Announced
is consistent with the client-application store, but published sounds
more fitting for events.
### Publishing and marking events as published
The current implementation fetches all events that haven't been marked
as announced, sets them as announced, and then emits them. It's possible
that Unleash would crash in the interim or something else might happen,
causing the events not to get published. Maybe it would make sense to
just fetch the events and only mark them as published after the
announcement? On the other hand, that might get us into other problems.
Any thoughts on this would be much appreciated.
This PR reuses the revision Id information from the "optimal 304 for
server SDKs" to improve the freshness of the frontend API config data.
In addition it allows us to reduce the polling (and eventually remove it
when we are confident).
---------
Co-authored-by: Gastón Fournier <gaston@getunleash.io>
## About the changes
- Introducing ISegmentService interface to decouple from the actual
implementation
- Moving UpsertSegmentSchema to OSS to be able to use types
- Added comments where our code is coupled with segments just to
highlight and have a conversation about some use cases if needed, but
they can be removed before merging
- Removed segment service from some project features as it was not used
This PR takes the project status API a step further by adding the
capability of providing a date to control the selection. We are
currently making calculations based on a gliding 30 day window, updated
once a day. The initial database structure and method for updating the
UI is outlined in this PR.
This PR moves the getProjectOverview method out from the project health
controller. It doesn't make sense that this method lives here anymore,
as over time it has grown into method that relays all information about
a single project. It makes more sense that this now lives on the root of
the project api. Also removes unwanted duplication of getProjectOverview
from the project-service and the project-health-service.
## About the changes
Introduce a snapshot version of instanceStats inside
instance-stats-service to provide a cached state of the statistics
without compromising the DB.
### Important notes
Some rule-of-thumb applied in the PR that can be changed:
1. The snapshot refresh time
2. The threshold to report appName with the metrics
## Discussion points
1. The snapshot could be limited to just the information needed (things
like `hasOIDC` don't change until there's a restart), to optimize the memory usage
3. metrics.ts (used to expose Prometheus metrics) has a [refresh
interval of
2hs](2d16730cc2/src/lib/metrics.ts (L189-L195)),
but with this implementation, we could remove that background task and
rely on the snapshot
4. We could additionally update the snapshot every time someone queries
the DB to fetch stats (`getStats()` method), but it may increase
complexity without a significant benefit
Co-authored-by: Mateusz Kwasniewski <kwasniewski.mateusz@gmail.com>
Co-authored-by: Simon Hornby <liquidwicked64@gmail.com>
In this PR we remove the general SettingService cache, as it will not
work across multiple horizontal unleash instances, events are not
published across.
We also fix the CORS origin to:
- Access-Control-Allow-Origin set to "*" if no Origin is configured
- Access-Control-Allow-Origin set to "*" if any Origin is configured to
"*"
- - Access-Control-Allow-Origin set to array and have the "cors"
middleware to return an exact match on the user provided Origin.
Co-authored-by: Fredrik Oseberg <fredrik.no@gmail.com>
Adds ability to favorite projects.
1. Can favorite project
2. Can unfavorite project
3. Favorite field is returned on `/api/admin/projects/default`
4. Favorite field is returned on` /api/admin/projects`
<!-- Thanks for creating a PR! To make it easier for reviewers and
everyone else to understand what your changes relate to, please add some
relevant content to the headings below. Feel free to ignore or delete
sections that you don't think are relevant. Thank you! ❤️ -->
This PR sets up exports so that we can import in enterprise with just
"unleash-server".
This will free us to refactor unleash internals without breaking
enterprise
## About the changes
<!-- Describe the changes introduced. What are they and why are they
being introduced? Feel free to also add screenshots or steps to view the
changes if they're visual. -->
<!-- Does it close an issue? Multiple? -->
Closes #
<!-- (For internal contributors): Does it relate to an issue on public
roadmap? -->
<!--
Relates to [roadmap](https://github.com/orgs/Unleash/projects/10) item:
#
-->
### Important files
<!-- PRs can contain a lot of changes, but not all changes are equally
important. Where should a reviewer start looking to get an overview of
the changes? Are any files particularly important? -->
## Discussion points
<!-- Anything about the PR you'd like to discuss before it gets merged?
Got any questions or doubts? -->
* fix: filter empty metrics before we collect last seen toggles.
fixes: #2104
* fix: add a last-seen service to batch last-seen toggle updates
Co-authored-by: Fredrik Strand Oseberg <fredrik.no@gmail.com>
* First version ready
* Final
* Refactor
* Update pat store
* Website revert
* Website revert
* Update
* Revert website
* Revert docs to main
* Revert docs to main
* Fix eslint
* Test
* Fix table name
* PublicSignupTokens
* bug fix
* bug fixes and test
* bug fixes and test
* bug fixes and test
* Add feature flag
* tests
* tests
* Update 20220908093515-add-public-signup-tokens.js
Bug Fix
* task: use swc instead of ts-jest (#2042)
* Add a counter for total number of environments (#1964)
* add groupId to gradual rollout template (#2045)
* add groupId to gradual rollout template
* FMT
* Improve tabs UI on smaller devices (#2014)
* Improve tabs UI on smaller devices
* Improve tabs UI on smaller devices
* bug fix
* add proper scrollable tabs
* removed centered from Tabs (conflicts with scrollable)
* PR comments
* 4.15.0-beta.10
* Fix broken doc links (#2046)
## What
This PR fixes some broken links that have been hanging around in the
docs for what seems like a very long time.
## Why
As discovered by the link check in #1912, there are a fair few broken
links in the docs. Everyone hates broken links because it makes it
harder to understand what they were supposed to be pointing at.
## How
There are 3 types of links that have been fixed:
- Links that should have been internal but were absolute. E.g.
`https://docs.getunleash.io/path/article` that should have been
`./article.md`
- External links that have changed, such as Slack's API description
- GitHub links to files that either no longer exist or that have been
moved. These links generally pointed to `master`/`main`, meaning
they are subject to change. They have been replaced with permalinks
pointing to specific commits.
-----
* docs: fix slack api doc link
* docs: update links in migration guide
* docs: fix broken link to ancient feature schema validation
* docs: update links to v3 auth hooks
* docs: update broken link in the go sdk article
* Fix: use permalink for GitHub link
* docs: fix wrong google auth link
* 4.15.0
* 4.15.1
* docs: update link for symfony sdk (#2048)
The doc link appears to have pointed at an address that is no longer reachable. Instead, let's point to the equivalent GitHub link
Relates to and closes#2047
* docs: test broken links in website (#1912)
The action triggers manually as a first step to test this functionality. In the near future, we might schedule it
* Schedule link checker action (#2050)
Runs at 12:30 UTC on Mon, Tue, Wed, Thu and Fri
* fix: add env and project labels to feature updated metrics. (#2043)
* Revert workflow (#2051)
* update snapshot
* PR comments
* Added Events and tests
* Throw error if token not found
Co-authored-by: Christopher Kolstad <chriswk@getunleash.ai>
Co-authored-by: Thomas Heartman <thomas@getunleash.ai>
Co-authored-by: Gastón Fournier <gaston@getunleash.ai>
Co-authored-by: Ivar Conradi Østhus <ivarconr@gmail.com>
Co-authored-by: sjaanus <sellinjaanus@gmail.com>
* refactor: remove unused API definition routes
* feat: add support for proxy keys
* feat: support listening for any event
* feat: embed proxy endpoints
* refactor: add an experimental flag for the embedded proxy
* Feat: return reasons why a feature evaluated to true or false
Note: this is very rough and just straight ripped from the nodejs
client. It will need a lot of work, but is a good place to start
* Feat: add suggested shape for new payload
* Chore: minor cleanup
* Wip: make server compile again
* Remove unused schema ref
* Export new schemas
* Chore: fix some tests to use sub property
* Fix: fix some tests
* Refactor: rename some variables, uncomment some stuff
* Add segments type to bootstrap options
* Add segments capability to offline feature evaluator
* Fix function calls after turning params into an option abject
* Feat: test strategy order, etc
* Feat: add test to check that all strats are returned correctly
* Feat: allow you to include strategy ids in clients
* Wip: hook up segments in the offline client.
Note: compared to regular clients, they still fail
* Feat: add segments validation
* Fix: fix test case invariant.
* Chore: revert to returning only `boolean` from strategies.
This _should_ make it work with custom strategies too 🤞
* Feat: make more properties of the returned feature required
* Wip: add some comments and unfinished tests for edge cases
* Feat: add `isEnabledInCurrentEnvironment` prop
* Feat: consider more strategy failure cases
* Feat: test that isenabledinenvironment matches expectations
* Feat: add unknown strategies
* Fix: fix property access typo
* Feat: add unknown strategy for fallback purposes
* Feat: test edge case: all unknown strategies
* Feat: add custom strategy to arbitrary
* Feat: test that features can be true, even if not enabled in env
* Chore: add some comments
* Wip: fix sdk tests
* Remove comments, improve test logging
* Feat: add descriptions and examples to playground feature schema
* Switch `examples` for `example`
* Update schemas with descriptions and examples
* Fix: update snapshot
* Fix: openapi example
* Fix: merge issues
* Fix: fix issue where feature evaluation state was wrong
* Chore: update openapi spec
* Fix: fix broken offline client tests
* Refactor: move schemas into separate files
* Refactor: remove "reason" for incomplete evaluation.
The only instances where evaluation is incomplete is when we don't
know what the strategy is.
* Refactor: move unleash node client into test and dev dependencies
* Wip: further removal of stuff
* Chore: remove a bunch of code that we don't use
* Chore: remove comment
* Chore: remove unused code
* Fix: fix some prettier errors
* Type parameters in strategies to avoid `any`
* Fix: remove commented out code
* Feat: make `id` required on playground strategies
* Chore: remove redundant type
* Fix: remove redundant if and fix fallback evaluation
* Refactor: reduce nesting and remove duplication
* Fix: remove unused helper function
* Refactor: type `parameters` as `unknown`
* Chore: remove redundant comment
* Refactor: move constraint code into a separate file
* Refactor: rename `unleash` -> `feature-evaluator`
* Rename class `Unleash` -> `FeatureEvaluator`
* Refactor: remove this.ready and sync logic from feature evaluator
* Refactor: remove unused code, rename config type
* Refactor: remove event emission from the Unleash client
* Remove unlistened-for events in feature evaluator
* Refactor: make offline client synchronous; remove code
* Fix: update openapi snapshot after adding required strategy ids
* Feat: change `strategies` format.
This commit changes the format of a playground feature's `strategies`
properties from a list of strategies to an object with properties
`result` and `data`. It looks a bit like this:
```ts
type Strategies = {
result: boolean | "unknown",
data: Strategy[]
}
```
The reason is that this allows us to avoid the breaking change that
was previously suggested in the PR:
`feature.isEnabled` used to be a straight boolean. Then, when we found
out we couldn't necessarily evaluate all strategies (custom strats are
hard!) we changed it to `boolean | 'unevaluated'`. However, this is
confusing on a few levels as the playground results are no longer the
same as the SDK would be, nor are they strictly boolean anymore.
This change reverts the `isEnabled` functionality to what it was
before (so it's always a mirror of what the SDK would show).
The equivalent of `feature.isEnabled === 'unevaluated'` now becomes
`feature.isEnabled && strategy.result === 'unknown'`.
* Fix: Fold long string descriptions over multiple lines.
* Fix: update snapshot after adding line breaks to descriptions
* Implement user grouping feature for permissions
Co-authored-by: Thomas Heartman <thomas@getunleash.ai>
Co-authored-by: Jaanus Sellin <sellinjaanus@gmail.com>
Co-authored-by: Nuno Góis <github@nunogois.com>
Co-authored-by: Thomas Heartman <thomas@getunleash.ai>
* Chore: add unleash-client dependency
* Feat: add openapi specs for basic sdk contexts and playground reqs
* Feat: add openapi spec for response
* Feat: add openAPI entry for playground endpoint
* Fix: remove required 'projects' property that doesn't exist
* Refactor: add test placeholders for playground api
* Chore: improve openAPI description for playground request
* Refactor: align example project names
* Fix: update openapi snapshot
* Fix: summary -> description; example -> examples
* Feat: add fast-check and stub out one test.
The test doesn't work yet.
* Feat: add test and generator for sdk context schema
* Feat: add generators and tests for all new schemas
* Feat: test that the same input is returned exactly
* Feat: add more tests and more schemas
* Refactor: clean up tests some
* Refactor: simplify url friendliness check
* Refactor: fix types for playground request schema generator
* Feat: add bad request test
* Fix: make parameters the correct shape in generator
* Feat: scaffold out tests for service and endpoint
Liberal use of @ts-expect-error
* Fix: update snapshot with new model
* Add missing appName
* wip: stub out some extra tests
* Feat: model strategies for testing
* Feat: model variant payloads better
* Refactor: input params for function
* Refactor: use email address for user id generation
* Chore: update comment
* Wip: start stubbing out e2e tests
* Refactor: move tests requiring db to e2e file
* Fix: make e2e tests run as expected
* Fix: make toggle generator test work as expected
* Fix: increase timeout for long-running e2e test
* Wip: map toggles test
* Wip: test movement and cleanup
* Refactor: remove `AllowedStrings` in favor of typeof X[number]
* Wip: add e2e tests for the service
* Wip: tests in progress
* Feat: return fuller variant value
* Wip: delete redundant test.
Both those tests were testing versions of the same thing, so let's
delete one of them
* Feat: update openAPI snapshot
* Feat: hook up the playground service
* Feat: fix test cases for e2e api
* Refactor: get rid of _one_ 'as unknown as x' cast
* Fix: wait for the client to be ready before we pass it back
* Fix: weights go to 1000!
* Fix: fix some test assertions, add one more test to be sure
* Wip: add some test parameters to hopefully avoid timeouts
* Fix: don't init the client if there are no toggles
If you don't provide it with any toggles to bootstrap, then the client
will never be ready. This avoids that.
This adds two ways around it because we'll probably get away with a
bit less computation if we don't even try to initialize it. However,
we also don't want anyone else to make this mistake.
* Chore: modify test params
* Feat: use the type system to avoid empty lists for initialization
* Refactor: make tests simpler
* Feat: test basic context values
* Feat: test dynamic context values
* Fix: make custom context field test more reliable
* Feat: use unique arrays for toggles
* Feat: make `appName` required in context
* Fix: sdk context schema arbitrary: make appName at least 1 char long
* Fix: add a minimum length on appName in the context schema
* Fix: fix tests that fail due to new requirements
* Fix: update a test: remove variant mapping testing
* Fix: fix remaining tests
* Feat: rename response.toggles -> response.features
* Refactor: avoid single-letter predicate variables
* Refactor: extract offlineclient into its own file
* Refactor: move arbitraries into separate file
* Refactor: move playground requests into util function
* Fix: remove duplicate declaration of urlFriendlyString
* Chore: update comment
* Refactor: align on arbitrary generator naming
* Feat: Export single strategy arbitrary
* feat: add OpenAPI validation to a few endpoints (2)
* refactor: use package version as the OpenAPI version
* refactor: keep the existing OpenAPI page for now
* refactor: add snapshots tests for the OpenAPI output
* refactor: validate Content-Type by default
* refactor: update vulnerable deps
* refactor: fix documentation URL to match schema
* refactor: improve external type declaration
* refactor: remove unused package resolutions
* refactor: try express-openapi fork
* Update package.json
* Update src/lib/services/openapi-service.ts
* Update src/lib/types/openapi.d.ts
* Update src/lib/types/openapi.d.ts
Co-authored-by: Ivar Conradi Østhus <ivarconr@gmail.com>
