Mirrors/unleash.unleash
1
0
Fork 0
mirror of https://github.com/Unleash/unleash.git synced 2025-05-26 01:17:00 +02:00
Code Issues Projects Releases Wiki Activity

docs: updates to 11 principles and scaling pages (#9688)

Browse Source
This commit is contained in:
Melinda Fekete 2025-04-09 14:35:43 +02:00 committed by GitHub
parent d5af58ce0e
commit 09c3f1ab40
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
9 changed files with 259 additions and 330 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

447
website/docs/topics/feature-flags/best-practices-using-feature-flags-at-scale.mdx
View File

@ -1,254 +1,123 @@
---
title: 'Feature flag management: Best practices for using feature flags at scale'
description: 'A feature flag is just an if statement, you say. When your organization has thousands of developers, those if statements can quickly become complex.'
title: 'Feature flag management: Best practices'
description: 'A guide for feature flag management. Best practices for organization, lifecycle management, and avoiding common pitfalls to keep your system efficient and secure.'
toc_max_heading_level: 2
---
“A feature flag is just an if statement,” you say. This is true in a way, but when your organization has thousands of developers, each managing dozens or hundreds of flags with complex targeting rules and audit trails across dozens of microservices-based applications, those if statements can quickly become complex. This is especially true if you don’t set up your system correctly in the first place.
# Feature flag management at scale: best practices
This guide to using feature flag systems at scale is based on lessons learned working with some of the largest feature flag deployments in the world through the [Unleash Open-Source](https://github.com/Unleash/unleash) and [Enterprise Feature Management platform](https://www.getunleash.io/enterprise-feature-management-platform). However, the principles outlined do not apply only to Unleash. They apply to any large-scale feature flag system, including one you build yourself or another commercial solution. For tips on how to stand up and run a feature flag system, see our [principles for building and scaling feature flags](./feature-flag-best-practices).
You've probably heard someone say, “A feature flag is just an if statement.” And in simple cases, that's true. But when your organization has thousands of developers managing hundreds of [feature flags](https://docs.getunleash.io/what-is-a-feature-flag)—with complex targeting rules, audit trails, and dozens of microservices—those if statements get complicated fast. Especially if you don't [set things up properly from the start](../../feature-flag-tutorials/use-cases/user-management-access-controls-auditing).
This guide is organized into three sections that align with the stages of a large-scale feature flag rollout:
This guide shares what we've learned from helping teams run some of the largest feature flag deployments in the world, both through [Unleash Open-Source](https://github.com/Unleash/unleash) and [Unleash Enterprise](https://www.getunleash.io/enterprise-feature-management-platform).
- [Organizing your team and internal processes to support feature flags](#organizing-your-team-and-internal-processes-to-support-feature-flags)
- [1. Align feature flags with organizational and application design.](#1-align-feature-flags-with-organizational-and-application-design)
- [2. Make flags searchable globally.](#2-make-flags-searchable-globally)
- [3. Design for flag permissions to change over time.](#3-design-for-flag-permissions-to-change-over-time)
- [4. Get flag management permissions right. And audit everything.](#4-get-flag-management-permissions-right-and-audit-everything)
- [5. Base flag permissions on global permissions.](#5-base-flag-permissions-on-global-permissions)
- [6. Implement flag approval workflows early.](#6-implement-flag-approval-workflows-early)
- [Instrumenting your code and managing flag lifecycle](#instrumenting-your-code-and-managing-flag-lifecycle)
- [7. Define flags in your code at the highest level of abstraction.](#7-define-flags-in-your-code-at-the-highest-level-of-abstraction)
- [8. Evaluate a feature flag only once.](#8-evaluate-a-feature-flag-only-once)
- [9. Continually pay down feature flag tech debt.](#9-continually-pay-down-feature-flag-tech-debt)
- [10. Leverage flag lifecycles to optimize your development](#10-leverage-flag-lifecycles-to-optimize-your-development)
- [Common pitfalls to avoid when implementing flags at scale](#common-pitfalls-to-avoid-when-implementing-flags-at-scale)
- [11. Avoid feature flag parent-child dependencies.](#11-avoid-feature-flag-parent-child-dependencies)
- [12. Don’t use flags to manage configuration.](#12-dont-use-flags-to-manage-configuration)
- [13. Never reuse feature flag names.](#13-never-reuse-feature-flag-names)
- [14. Avoid giant feature flag targeting lists.](#14-avoid-giant-feature-flag-targeting-lists)
- [15. Do not define business logic using feature flags.](#15-do-not-define-business-logic-using-feature-flags)
- [Dependency on 3rd party services](#dependency-on-3rd-party-services)
- [Complexity and Maintainability](#complexity-and-maintainability)
- [Performance Implications](#performance-implications)
- [Security Risks](#security-risks)
- [Keeping cool while managing feature flags at scale](#keeping-cool-while-managing-feature-flags-at-scale)
The principles here don't only apply to [Unleash](https://www.getunleash.io/guide-to-feature-management)—they're relevant for any large-scale feature flag system, whether you're building your own or using a commercial solution. If you're just getting started, check out our [best practices for building and scaling feature flags](./feature-flag-best-practices).
## Organizing your team and internal processes to support feature flags
In this guide, we'll cover tips and best practices in five key focus areas to help you scale a feature flag system:
Before you add your first feature flag, you need to think about how organizational structure and processes that live outside of code will affect your feature flag deployments.
- [Organize your feature flag code effectively](#organize-your-feature-flag-code-effectively)
- [Stay on top of technical debt](#stay-on-top-of-technical-debt)
- [Use feature flag lifecycle to optimize your development workflow](#use-feature-flag-lifecycle-to-optimize-your-development-workflow)
- [Avoid common pitfalls when implementing feature flags](#avoid-common-pitfalls-when-implementing-feature-flags)
- [Enable secure collaboration with flexible permissions](#enable-secure-collaboration-with-flexible-permissions)
### 1. Align feature flags with organizational and application design.
## Organize your feature flag code effectively
> Any organization that designs a system will produce a design whose structure is a copy of the organization's communication structure.
Feature flags live in your codebase—and how you implement them matters. The choices you make when organizing flags can impact performance, testability, and long-term maintainability. Here are a few best practices we've learned from working with large-scale systems.
– Melvin Conway
### Define flags at the highest level of abstraction
![Feature flag organization methods](/img/feature-flag-organization-methods.svg)
Place feature flag evaluations as high in the stack as possible. For example, when testing a dark mode toggle or a new signup flow, evaluate the flag near the UI layer rather than scattering it across multiple components. This helps you:
- **Simplify your code**: Control feature behavior from a single point so that the rest of your system does not need to be aware of the flag.
- **Test more easily**: Isolate the feature for unit and integration testing.
- **Clean up faster**: Flags defined in one place are easier to find, maintain, and clean up when no longer needed.
There is no getting around [Conway’s law](https://martinfowler.com/bliki/ConwaysLaw.html). Applications tend to resemble the organizations that create them, and your feature flag system is no exception. Rather than fighting this law, acknowledge and embrace it, and pick a way to organize your feature flags that reflects your organization.
![Evaluate flags at the edge](/img/evaluate-flags-at-the-edge.jpg)
A typical Fortune 500 company typically has thousands of software developers. Organizing your feature flag system based on “applications” when your company is organized based on business units or products will lead to a poor experience and additional complexities in managing flag permissions.
Users should be shown information about any flags related to their work, whether on a team, an application, a product, or a project basis. This includes inspecting the flag’s configuration and tracking updates to the flag.
Achieve this by grouping related flags together based on the most appropriate level of abstraction. If you work on static teams that see a feature through from start to finish, group flags by team. If you form project-based teams, group flags by project.
Taken to the extreme, you might be tempted to put all flags into one giant group. Avoid that temptation because it will create permission complexities and information overload.
In an organization that actively uses feature flags, there will invariably be a large number of flags and an even larger number of changes made to those flags. This can quickly become overwhelming and a user may end up losing track of updates that are important to them.
Removing information and updates you don’t care about makes the things you care about stand out that much more.
**Example**: Do all members of Team A need updates on Project X, even if only one member of Team A is working on it? If so, include all team members in Project X’s group. Otherwise, only include the member who works on project X.
### 2. Make flags searchable globally.
Modern applications are composed of multiple services with many complex dependencies. While you should try to organize your flags based on high-level abstractions such as applications, teams, or business units, many services cut across these artificial organizing principles.
Just because a user might need to interact with a flag does not mean you should display it by default when it is outside your main organizational parameters. However, all flags should be easily searchable. When a user finds one, they should be able to inspect its configuration and ownership so that they might request additional permissions or [submit change requests for approval](#6-implement-flag-approval-workflows-early).
This is why feature flag systems should be [open by default](./feature-flag-best-practices#5-choose-open-by-default). There are valid use cases for excluding flags from global search. For example, a public company in the middle of an acquisition where exposing flags related to the website or app changes might breach regulatory guidelines. Your feature flag system should accommodate these private projects; however, they should be the exception, not the rule.
### 3. Design for flag permissions to change over time.
The owner or maintainer of a flag may change over time as the feature it controls [evolves across its lifecycle](#10-leverage-flag-lifecycles-to-optimize-your-development).
For instance, when a developer starts working on a feature, they’ll place it behind a flag so that they can merge back into the main branch early and often (a development style known as [trunk-based development](https://www.getunleash.io/blog/how-to-implement-trunk-based-development-a-practical-guide#What%20is%20trunk-based%20development)) without affecting the product. At this point, the flag exists to hide unfinished code and should be owned and managed by the developer.
When the feature is ready to be rolled out, the flag’s purpose will change. Its purpose is no longer to hide unfinished code but rather to _expose_ new functionality. This can be done in several ways and will vary from feature to feature.
If the feature is a simple improvement, the change can likely be incrementally rolled out while keeping an eye out for any signs that the feature is misbehaving. This can be managed entirely by the developer.
However, if the feature rollout has certain requirements, it’s quite likely that the developer doesn’t have the entire context that they need. As such, the ownership is now shared between the developer and a product owner. In cases like this, the developer may be in charge of the main switch, enabling or disabling the feature so that they can turn it off if something goes wrong. However, the product owner might decide what targeting strategies to use, such as how many users to roll the feature out to.
This is also often the case for B2B companies where specific customers request certain functionality. A Customer Success Manager will often have better insight into to whom to roll out a new feature first.
To be clear, the developers should still be kept in the loop (after all, when was the last time your Customer Success Managers checked system logs?). However, the flags are now a shared responsibility, and other team members should be given the power and responsibility to make the changes they want to see.
Here are common examples that require updating flag permissions you should plan for:
- Permissions are wrongly scoped. Someone was left out, or someone who should not have access does.
- Requirements changed mid-project.
- The project owner or team lead changed.
- Teams are reorganized.
The flag lifecycle stage dictates changes in the owners.
### 4. Get flag management permissions right. And audit everything.
It is clear now that different teams will require access to flags across their lifecycle. However, not every user will need the same permissions for common flag management tasks.
Here is a partial list of things one or more users will want to do with a feature flag for which you need to configure permissions:
- Create
- Delete
- Turn-on
- Turn-off
- Configure targeting
- Change rollout
- Read configuration
- Update configuration
Let’s look at a simple example: a product manager working with the development team to test a new Beta feature.
The developer should most likely be able to turn the flag on and off. Should the product manager? That depends on your organization's culture and requirements. Your feature flag system should be able to deal with this permission complexity.
The same thing goes for editing targeting rules (i.e. who sees the version of the application controlled by the flag) and rollout strategies (i.e., what % of users are exposed to the feature). It is up to your organization how much the product owner can do independently. It might make sense for them to be able to update and edit targeting rules for a flag freely but not control the rollout itself. Or, they might be able to make the edits to targeting and rollout directly but need to submit a change request for their desired changes and have them reviewed by a manager or more technical stakeholder. These permissions are directly related to an organization's security and compliance policies and are important to get right.
Equally important is auditing all of these permissions and changes. As such, knowing where to find the flagging system’s audit log is essential. If there’s a slip-up, you can find out when and where it happened and use those findings to improve your processes. Should you require more approvals before going into production? Do you need to update targeting rules or user segments? In the case of a malicious actor, the audit log can help you identify the actor and the changes they made.
### 5. Base flag permissions on global permissions.
Your organization has already established some notions of groups. Leverage that for your feature flag system permissions.
These might be groups based on roles (such as developers, operations, marketing, etc.), or business units, or both.
These groups should be mirrored in your feature flag system to help you organize your users and access rights. Your system should be set up to sync with SSO providers to make this process as painless as possible.
Users who are added or removed from your main SSO directory should be added or removed (based on their role) from your feature flag system. Managing global permissions is complex. Only do it in one place.
### 6. Implement flag approval workflows early.
![Change request process](/img/change-request-flow.png)
Depending on the industry and legal framework you’re operating in, you’ll need varying approvals for a feature flag change to go into production. You may be able to just roll something out on a whim, or you might need to get it approved by specific stakeholders before releasing it. Your feature flag tool should help you do this, regardless of how much oversight you need.
**Require approvals where necessary**
When legal requirements say that a change must be peer-reviewed before being put into production (often referred to as the ‘[four-eyes principle](https://www.unido.org/overview-member-states-change-management-faq/what-four-eyes-principle)' or the ‘two-person rule’), it should be easy to do. You should be able to add all the changes you want to make to all the relevant flags and present them as a group, similar to making a pull request on GitHub.
It should be easy for the reviewers to see what changes would be made and what the configuration would look like after the change.
But this isn’t just useful for legal reasons. The process of submitting something for review is useful in and of itself.
Review flag changes before they take effect
Even if you don’t require anyone else's approval before making changes in production environments, there are still benefits to grouping your changes together and reviewing them before actually making them:
1. You can make multiple changes at the same time. If you have a set of changes across flags that should change in tandem, you can make sure they update at the same time, avoiding awkward in-between states.
2. You avoid making changes by mistake. If your cat walks across the keyboard, you can feel safe that they won’t accidentally expose your unfinished code to everyone.
3. You get to review the changes before they go out. You get the full overview of the changes you’re making. You can double-check that the changes will have the effect you want.
## Instrumenting your code and managing flag lifecycle
Flags live in code, and hundreds of small choices go into instrumenting flags into that code. Here are some tips we’ve picked up to improve the maintainability and performance of large-scale feature flag systems.
### 7. Define flags in your code at the highest level of abstraction.
![Evaluate flags at the edge](/img/evaluate-flags-at-the-edge.svg)
Defining your flag at the highest level in your code is important for both user-facing and backend changes.
For user-facing changes (e.g., testing a light/dark mode or a new signup workflow), instead of evaluating a single feature flag in multiple locations in your code, try to evaluate the flag close to the user interface (UI) – "far out in the stack." This allows for more granular control based on user context and eliminates the need for flags in lower-level parts of the system.
Instrumenting your code in this way has several benefits:
- Simplified Code: By controlling a new feature at a single location, the rest of your system doesn't need to be aware of the flag, leading to cleaner code.
- Enhanced Testing: Easier isolation of the feature for unit and integration tests.
- Improved maintainability: When a feature flag is used at a single location, it is easier to clean up the flag when it has served its purpose. In addition, simpler and testable code is easier to maintain over time.
For backend changes (e.g., a new database schema), evaluating the feature flag near the affected module is best.
For backend logic—such as introducing a new database schema or optimizing a query—evaluate the flag close to the affected module. Make sure your code is easily testable. Evaluate the flag externally and inject the result, or provide two separate implementations. This makes it easier to write unit tests for both versions of the logic.
This enables:
- Focused Testing: Evaluating the flag's impact within the specified module simplifies testing and debugging. You only need to consider the immediate code's behavior, making troubleshooting more efficient.
- Focused testing: Evaluating the flag's impact within the specified module simplifies testing and debugging. You only need to consider the immediate code's behavior, making troubleshooting more efficient.
- Isolation: Evaluating close to the module helps isolate the impact of the feature flag, reducing the risk of unintended consequences in other parts of the codebase.
- Testable: Make sure your modules are testable. Usually, it makes sense to evaluate the flag logic outside and either inject the result or simply have two different implementations of the module. This optimizes for testability, making it straightforward to implement unit tests for both the old and the new logic.
Once you have defined your flags, striving for single evaluation points whenever possible is crucial. This reduces code complexity and potential errors from managing multiple evaluation points.
:::note
Here are a few examples of functionality managed by feature flags and where you should try to evaluate in your code:
Whenever possible, evaluate your flag in a single place. Centralizing the evaluation simplifies your code and reduces the chances of inconsistent behavior.
- Recommendation Engine: When testing changes to the recommendation algorithm, evaluating the feature flag close to the recommendation module allows focused testing and isolation of the changes.
- Database Query Optimization: Evaluating the flag near the optimized query simplifies testing and ensures the change behaves as expected within that specific query.
- Third-Party Integrations: If a feature flag controls the integration with a third-party service, evaluating it near the integration code helps isolate the behavior changes. This simplifies testing the interaction with the third-party service based on the flag state, without affecting other integrations in the system.
- Cache Invalidation Logic: When modifying code that invalidates cache entries based on a feature flag, evaluating the flag near that logic allows for focused testing of the cache invalidation behavior. This isolates the test from other parts of the system that could potentially interact with the cache.
:::
### 8. Evaluate a feature flag only once.
### Evaluate feature flags once per user request
![Evaluate flags once](/img/evaluate-flags-only-once.svg)
When rolling out complex features, changes often span multiple parts of a system—whether that's modules within a single application or services in a distributed architecture. It might seem convenient to use the same feature flag across all components and evaluate it locally in each one, but this can lead to problems.
When building new (often complex) features for users, systems require changes across multiple parts – modules within an application or services in a microservices architecture. While it's tempting to use a single feature flag to control all these changes and evaluate it locally in each module, we recommend against it for a few reasons:
As a user request moves through the system, it hits different parts at different times. Even with synchronized flag states, there's a risk that the flag's value changes between evaluations, resulting in inconsistent behavior for the user.
Particularly in distributed systems, we cannot assume all parts of the system are perfectly synchronized, as networks are unreliable and can experience errors at any time. Most feature flag systems prioritize [availability over consistency](./feature-flag-best-practices#2-prioritize-availability-over-consistency). By only evaluating a feature flag once, we guarantee a consistent experience for our users.
- Unpredictable Timing: User requests flow through the system and touch different parts as they progress. Between each invocation, time passes. Even with perfectly synchronized feature flags, requests will hit different system parts at slightly different times. This means the flag state could change between evaluations, leading to inconsistent behavior for the user.
- Laws of distributed computing: Particularly in distributed systems, we cannot assume all parts of the system are perfectly synchronized, as networks are unreliable and can experience transient errors at any time. Feature flag systems generally [prefer availability over consistency](./feature-flag-best-practices#2-prioritize-availability-over-consistency). By only evaluating a feature flag once, we guarantee a consistent experience for our users.
- Violates Single Responsibility: Using the same flag in multiple locations spreads feature control logic throughout your codebase, violating the single-responsibility principle.
Using the same flag in multiple places fragments the control logic and breaks the single responsibility principle, making code [harder to understand and maintain](https://www.getunleash.io/simplicity).
We find that companies are usually more successful when using feature flags when they can protect new complex features with a single flag evaluated only once per user request. This approach ensures consistent behavior and simplifies long-term code maintainability.
![Evaluate flags once](/img/evaluate-flags-in-many-places.jpg)
### 9. Continually pay down feature flag tech debt.
Instead, evaluate the flag once—typically at the start of a user request—and pass the result through the system. This keeps behavior predictable and makes the codebase cleaner and easier to manage over time.
We love feature flags for the way that they increase developer productivity, reduce risk and enable data-driven product development. But feature flags are also technical debt. How can both be true? The answer lies not with flags but with how they are managed.
![Evaluate flags once](/img/evaluate-flags-only-once.jpg)
Feature flag technical debt accumulates when feature flags are not properly managed or retired after their intended use. Over time, the codebase becomes cluttered with outdated or unnecessary flags, making the code more complex and harder to maintain. This increased complexity can slow development productivity as developers spend more time understanding and navigating the code. Additionally, neglected feature flags can introduce security vulnerabilities if they unintentionally expose sensitive features or data. Furthermore, the presence of stale or conflicting feature flags can lead to unexpected app behaviors, increasing the risk of downtime and affecting overall application stability. Managing feature flags effectively minimizes these risks and maintains a healthy development workflow.
## Stay on top of technical debt
This is why we [avoid using feature flags for things like application configuration](#12-dont-use-flags-to-manage-configuration) for which they are not a great fit. More flags, more debt. Like any debt, it can be used efficiently to accomplish goals that would otherwise be difficult or impossible (e.g., buying a home). But left unchecked, you can end up in a world of pain.
[Feature flags are powerful](https://www.getunleash.io/blog/feature-ops-is-the-next-frontier)—they increase developer productivity, reduce risk, and support data-driven product decisions. But they also come with a catch: they're a form of [technical debt](../../reference/technical-debt).
Paying down technical debt–in its simplest form, removing old flags–requires an understanding of flag lifecycles and the ability to track them.
Technical debt builds up when flags stick around long after they've served their purpose. Over time, outdated or unused flags clutter your codebase, making it harder to read, test, and maintain. This slows down development and can even lead to bugs, [security risks](https://www.getunleash.io/security-and-performance), or unexpected behavior—especially if old flags conflict with one another or expose sensitive features.
### 10. Leverage flag lifecycles to optimize your development
The simplest way to pay down flag debt? Track your flags and remove them when they're no longer needed. Understanding the full lifecycle of a flag is key to keeping your codebase clean and your team moving fast.
All feature flags progress through a set of predefined lifecycle stages even if these stages are not explicitly defined in your feature flag system:
## Use feature flag lifecycle to optimize your development workflow
- Define: The feature flag is defined, but no code implementation has begun.
- Develop: Some code is implemented, but the feature is not enabled for any users in production. Internal testing and validation are ongoing.
- Production: The code is deployed to production and gradually rolled out to a controlled set of users for validation with real users.
- Cleanup: A team has decided to keep or cancel the feature. The feature flag still exists, but it is now time to clean up the code protected by it to ensure no more technical debt is acquired.
- Archived: The feature flag has reached its end-of-life and has been disabled. The code associated with the feature flag is cleaned up to avoid technical debt.
Every feature flag follows a [lifecycle](../../reference/feature-toggles#feature-flag-lifecycle)—even if your tooling doesn't make it explicit. Understanding and tracking this lifecycle helps teams reduce technical debt and identify bottlenecks in the development workflow.
By monitoring feature progression through the lifecycle stages, organizations gain valuable insights from which they can benefit to improve their development process and pay down technical debt. Some key benefits of tracking feature flag lifecycle include:
Here are the typical stages:
- **Define**: The flag is created, but no code has been written yet.
- **Develop**: Code is in progress. The feature isn't live, but it's being tested internally.
- **Production**: The feature is deployed to production and gradually rolled out to users.
- **Cleanup**: The team has decided to keep or discard the feature. The flag is still active, but it's time to clean up the code.
- **Archived**: The flag is no longer needed. It's been disabled and the associated code removed.
- Identifying Bottlenecks: Teams can analyze the average time spent in each stage and pinpoint bottlenecks. Some examples include:
1. Stuck in Define: This may indicate issues like unclear requirements or integration difficulties in pre-production environments.
2. Stuck in Develop: Suggest challenges in achieving production readiness, like incomplete testing or code bugs.
3. Stuck in Production: This could imply difficulties gathering data or making decisions about the feature's future (e.g., lack of clear success metrics).
4. Stuck in Cleanup: Signals delays in decommissioning the feature flag and cleaning up resources, potentially creating technical debt.
- Easily Remove Stale Feature Flags: Feature flags left active after their purpose is served tend to create technical debt. This code is unused, increasing the risk of accidental exposure in production and making the codebase more complex to maintain. By retiring flags promptly, developers understand which code sections can be safely cleaned up, reducing technical debt.
- Data-Driven Insights: By collecting and aggregating metrics across features and teams over time, organizations gain valuable insights into team performance trends. Suddenly, it is easy to measure time-to-production or the number of times we need to disable a feature in production. Over time, this level of insights allows organizations to identify gradual changes in team efficiency or identify areas needing improvement that might otherwise go unnoticed.
Tracking flags through these stages helps teams work smarter. Here's how:
## Common pitfalls to avoid when implementing flags at scale
**Spot bottlenecks**
- Stuck in Define? Maybe requirements aren't clear. Perhaps there are integration difficulties.
- Stuck in Develop? Could point to testing delays or bugs.
- Stuck in Production? Might be missing success criteria or user feedback.
- Stuck in Cleanup? Check for accumulating technical debt.
This section could have been called “Been There. Done That. Got the T-shirt.” We’ve seen it all when it comes to large-scale flag implementations, and here are some of the things that seem like a good idea at the time but come back to bite you at scale.
**Clean up stale flags**
### 11. Avoid feature flag parent-child dependencies.
Flags that outlive their purpose clutter the codebase and can lead to bugs or [security risks](https://www.getunleash.io/security-and-performance). Regularly reviewing the lifecycle data helps you keep your code clean and secure.
For larger organizations, it is often tempting to group multiple feature flags together and establish a “parent-child” dependency. The purpose of this relationship is to make it easier to coordinate releases. Often, you would like to have a single parent that will enable/disable multiple features that work together in concert.
**Get actionable insights**
Dependent feature flags make sense when you have different parts of a complex feature you want the option to disable individually but want the ability to control the gradual rollout strategies for each part of the feature together. The parent flag also acts as a global kill switch for the sibling feature flags.
Track metrics like time-to-production, rollback rates, or cleanup delays across teams. Over time, these insights can highlight efficiency gains—or uncover hidden slowdowns you might otherwise miss.
However, you should be careful when introducing these dependencies, as it increases the complexity of everyone interacting with the feature flag system. It's no longer enough to consider the configuration of a single flag, you also need to consider the configuration of the parent. It's easy to cause confusion, leading to accidentally enabling or disabling a feature in production
## Avoid common pitfalls when implementing feature flags
To keep complexity down, we generally advise not to have complex targeting rules for both the parent and the child flag simultaneously. This can make it harder to achieve consistency for end users. It also increases the complexity for the user of the feature flag system. For example, if both the parent and the child are configured to a 50% rollout, are you then exposing the feature to 50% or 25% of the user base? The answer is that it depends on the identifier used to roll the flag out.
When you work with feature flags at scale, some decisions seem smart in the moment—but can cause big headaches down the line. Here are five of the most common pitfalls we've seen:
### 12. Don’t use flags to manage configuration.
### Parent-child dependencies with complex targeting rules
Both feature flags and configuration settings control an application’s behavior. However, it's crucial to distinguish between feature flagging systems and configuration management systems and use each for its intended purpose. Otherwise, you risk management nightmares and security vulnerabilities
At scale, it's tempting to group related feature flags under a single parent flag to coordinate complex releases. The idea is simple: toggle one parent flag to enable or disable multiple child flags together. The parent can also serve as a global kill switch.
Feature flags are intended for dynamic and temporary use, giving you [runtime control](./feature-flag-best-practices#1-enable-runtime-control) to enable or disable functionality in a live environment. This flexibility supports [short-lived experiments](./feature-flag-best-practices#3-make-flags-short-lived), such as A/B testing or gradual rollouts, instead of relying on configuration settings that require an application redeploy to take effect. Despite the ease of turning off and on features or functionality with feature flags, they should not be repurposed for managing configuration.
While this might work in theory, in practice it adds significant complexity. Now, enabling or disabling a feature isn't just about a single flag—you have to account for both the parent and the child. It becomes easy to misconfigure rollouts or cause unintended behavior in production.
A dedicated config management system should manage long-term configuration settings. These systems are designed to handle sensitive information such as API keys and access credentials securely, unlike feature flag systems, which typically do not encrypt flag states. Misusing feature flags for configuration can lead to security risks and management complexities.
Things get even trickier when both parent and child flags have their own targeting rules. For example, if both are set to a 50% rollout, are you exposing the feature to 50% or 25% of users? The answer depends on how rollouts are calculated—which can be confusing and inconsistent.
A good rule of thumb is that if the data is static (you don’t expect it to change without restarting your application), needs encryption, or contains PII (Personal Identifiable Information), it’s probably better suited for your Configuration management system than a feature flag system.
Our advice: Keep it simple. If you use parent-child structures, avoid overlapping rollout rules, and document the hierarchy clearly. Better yet, try to design flags so that they can be managed independently.
### Using feature flags to manage configuration
Both feature flags and configuration settings control an application's behavior. However, it's crucial to distinguish between feature flagging systems and configuration management systems and use each for its intended purpose.
[Feature flags are temporary and dynamic](https://docs.getunleash.io/what-is-a-feature-flag). They give you [runtime control](./feature-flag-best-practices#1-enable-runtime-control) to enable or disable functionality in a live environment—perfect for things like [short-lived experiments](./feature-flag-best-practices#3-make-flags-short-lived), such as A/B testing or gradual rollouts.
Configurations, on the other hand, are usually static and long-lived. They often contain sensitive data like API keys or access credentials, which need to be encrypted and carefully managed. Using feature flags for configuration can create security vulnerabilities and operational issues. Most flag systems don't encrypt values the way configuration management tools do.
A good rule of thumb is that if the data is static (you don't expect it to change without application restart), needs encryption, or contains PII (personally identifiable information), manage it through a proper configuration system—not a feature flag.
<table>
<thead>
@ -264,7 +133,7 @@ A good rule of thumb is that if the data is static (you don’t expect it to cha
</tr>
<tr>
<td>
Examples of things to update with the config system:
Update with config systems:
<ul>
<li>Database or server credentials</li>
<li>Server port</li>
@ -274,7 +143,7 @@ A good rule of thumb is that if the data is static (you don’t expect it to cha
</ul>
</td>
<td>
Examples of things to update with feature flag system:
Update with feature flags:
<ul>
<li>Dark mode/light mode</li>
<li>Stable / Beta feature</li>
@ -286,58 +155,164 @@ A good rule of thumb is that if the data is static (you don’t expect it to cha
</tbody>
</table>
### 13. Never reuse feature flag names.
### Reusing feature flag names
Feature flag names need to be globally unique. [We wrote about this before](https://docs.getunleash.io/topics/feature-flags/unique-names), but it is such an important point that it bears repeating. Every feature flag within the same Feature Flag Control service must have a unique name across the cluster to prevent inconsistencies and errors.
Feature flag names need to be [globally unique](https://docs.getunleash.io/topics/feature-flags/unique-names). In an ideal world, all flag references would be removed from the codebase as soon as a flag is archived—but in reality, that rarely happens. Using unique names helps protect new features from accidentally linking to old, unused flags that could unintentionally re-enable outdated behavior.
In a perfect world, all references to feature flags in code would be cleaned up as soon as the feature flag is archived. But we don’t live in a perfect world. You will save yourself many hours of frustration and potential downtime if you use unique names to protect new features from being accidentally linked to outdated flags, which could inadvertently expose old features. [Unique names should ideally be enforced during flag creation](https://docs.getunleash.io/reference/feature-toggles#set-a-naming-pattern).
To avoid this risk, enforce a [naming pattern](https://docs.getunleash.io/reference/feature-toggles#set-a-naming-pattern) at flag creation.
### 14. Avoid giant feature flag targeting lists.
### Using large targeting lists
When implementing feature flags, it's crucial to aim for the quickest evaluation times possible. To achieve this, it's advisable to streamline the design of your constraints, which define when a feature protected by a flag is exposed.
When implementing feature flags, it's crucial to aim for the quickest evaluation times possible. To achieve this, avoid relying on large inclusion lists—like thousands of specific user IDs—which can slow things down and become difficult to maintain.
Avoid relying heavily on extensive lists of inclusion criteria, which can slow down the process. Instead, delve into the underlying factors determining whether a user qualifies for inclusion in these lists. For instance, consider users who have registered for a beta-testers program. An even more efficient approach would be to integrate an external registration system for the beta program and configure your feature flag system to recognize "beta" group members.
Instead, focus on the underlying traits that qualify users for a feature. For example, rather than building a large list of beta testers, use a user attribute that can be managed in an external system.
You can then configure your feature flag system to recognize the attributes that indicate that the user is part of the beta testing group.
This method of managing group memberships on a per-user basis significantly enhances the maintainability of your feature flagging system. Adding a new user to the beta testing group becomes a straightforward process. There's no need to manually update the configuration to ensure all existing user IDs are accounted for; simply enroll the new user in the beta program.
This approach helps with:
- Maintainability: Adding a new user to the beta group is as simple as updating their group membership, without needing to edit the flag configuration.
- Memory usage: Storing a simple attribute is much more efficient than holding a list of thousands of user IDs in memory.
- Compliance: Restricting personal data to only the systems that need it makes it easier to [stay compliant](../../using-unleash/compliance/compliance-overview).
Avoiding big lists also helps with memory usage since most well-designed feature flagging systems keep feature configuration in memory. A list of 10,000 user IDs will require more memory than keeping a true/false record of a user membership in a specific group.
### Defining core business logic in feature flags
Initially, using a list of specific handpicked users might work fine. But once you approach 100 users, maintaining this list becomes unwieldy. Once it does, you will probably keep this list in at least one other place. It would be better to keep this list updated in one place and use that to set a context field for the user. You could then use that field for feature flagging.
Feature flags [reduce the friction of releasing software](https://www.getunleash.io/feature-flag-use-cases-progressive-or-gradual-rollouts), so it's no surprise that teams who adopt them often expand their usage. This is generally a good thing—as long as archived flags are removed from the code base keeping tech debt down.
The last thing is that maintaining these lists also risks leaking PII to systems that don't need it. With PII, the fewer places you need to handle it, the easier it will be to make your systems compliant with frameworks such as GDPR/Schrems II or CCPA.
However, one common pitfall is using feature flags to control core business logic. Just like you shouldn't use feature flags to [manage application configuration](#using-feature-flags-to-manage-configuration), you should avoid embedding fundamental business rules in flags.
### 15. Do not define business logic using feature flags.
We recommend using feature flags to experiment with business logic, not to define it long-term. For example, let's say you want to test whether premium users will adopt a new feature. Wrapping it in a flag lets you test and measure safely. But once you've decided that all premium users should have access, remove the flag and move the logic into your entitlement service or backend code.
In our experience, since feature flags reduce the friction of releasing software, teams who adopt feature flags tend to use them more and more. This is a good thing as long as archived flags are removed from the code base keeping tech debt down. You should resist, however, the temptation to codify core business logic inside feature flags, the same way you should [resist wrapping application configuration in feature flags](#12-dont-use-flags-to-manage-configuration).
So, if feature flags are so useful, why not use them for business logic? Here's why:
Here is a good rule of thumb: Experiment with business logic using feature flags. But once you have established the business logic, codify it in code that lives outside of a flag.
#### Dependency on external services
For example, if you want to determine whether premium users of your product will use a certain new feature, you can wrap it in a flag and measure the result. Once you have determined that premium users do use the feature, and that all premium users should have access to it, however, you should remove the flag and add the feature to your entitlement service.
We don't recommend making core business logic dependent on any feature flag service. Even though Unleash is built with [high resilience](https://www.getunleash.io/blog/unleash-enterprise-edge-think-of-it-as-a-cdn-for-your-feature-flags) in mind—and designed to help your application keep running smoothly, even if you temporarily lose access to it—critical business functionality should never rely solely on the availability of an external service. This principle applies whether you’re using Unleash, another third-party provider, or a homegrown solution.
If feature flags are so great, why shouldn’t you use them for business logic? A few reasons:
#### Complexity and maintainability
#### Dependency on 3rd party services
Embedding business logic within feature flags can make the codebase unnecessarily complex. Business rules can become scattered and entangled with feature flagging logic, making the code harder to read, understand, and maintain. When changes are needed, developers might have to navigate through many feature flags to find and update the relevant logic, increasing the risk of introducing bugs.
Coming from a feature flag vendor, it might surprise you that we do not advocate making core business logic dependent on a 3rd party feature flag service. If that flag service is down, then your app could potentially cease to function in the way designed. This applies to using 3rd party feature flag services, as well as home grown feature flags service.
#### Performance implications
#### Complexity and Maintainability
Feature flags are typically designed to be checked frequently and quickly, with minimal performance overhead. However, when feature flags are used to control business logic, [they may involve more complex evaluations and data fetching, which can degrade application performance](#using-large-targeting-lists). This is particularly problematic for high-traffic applications where performance is critical.
Embedding business logic within feature flags can make the codebase unnecessarily complex. Business rules can become scattered and entangled with feature flagging logic, making the code harder to read, understand, and maintain. When changes are needed, developers might have to navigate through a labyrinth of feature flags to find and update the relevant logic, increasing the risk of introducing bugs.
#### Performance Implications
Feature flags are typically designed to be checked frequently and quickly, with minimal performance overhead. However, when feature flags are used to control business logic, [they may involve more complex evaluations and data fetching, which can degrade application performance](#14-avoid-giant-feature-flag-targeting-lists). This is particularly problematic for high-traffic applications where performance is critical.
#### Security Risks
#### Security risks
Business logic often involves access controls and entitlements. Using feature flags to manage these aspects can expose security vulnerabilities if not handled correctly. Feature flags might be toggled accidentally or maliciously, leading to unauthorized access or exposure of sensitive data.
## Keeping cool while managing feature flags at scale
## Enable secure collaboration with flexible permissions
These best practices come from working with many of the largest organizations on the planet. We’ve learned that keeping cool while managing feature flags at scale requires careful planning, thoughtful organization, and adherence to several best practices. By aligning feature flags with organizational structure, making flags searchable, and managing permissions effectively, you can maintain a clean and efficient system that sets you up for success from the start. Instrumenting your code correctly and regularly paying down technical debt helps maintain performance and security. Finally, avoiding common pitfalls, such as using feature flags for business logic or configuration management, keeps your system robust and manageable.
Before you add your first feature flag, you need to think about how organizational structure and processes that live outside of code affect your feature flag deployments. Also consider what legal and regulatory requirements you have—and how access, ownership, and collaboration is expected to change as your teams grow and your features evolve.
Feature flags are a powerful tool for delivering software efficiently while maintaining security and compliance. For developers working in large organizations, that is the best of both worlds. By following the best practices outlined in this guide, you can harness the full potential of feature flags while avoiding the complexities and risks associated with their misuse.
### Align feature flags with your organizational and application structure
Remember, the key to successful feature flag management at scale lies in clear processes, regular maintenance, and a commitment to best practices. With these in place, you can leverage feature flags to drive innovation and improve your software development lifecycle.
> Any organization that designs a system will produce a design whose structure is a copy of the organization's communication structure.
>
> – Melvin Conway
This guide was all about using feature flags at scale. For similar tips on building and scaling the feature flag system itself, see our [feature flag best practices guide](./feature-flag-best-practices).
[Conway's Law](https://martinfowler.com/bliki/ConwaysLaw.html) tells us that the systems we build naturally reflect the way our teams are structured—and that includes how we organize feature flags. Instead of working against this, it's best to acknowledge it and design your feature flag system in a way that mirrors your organization's structure.
For example, a large enterprise with thousands of developers might be structured around business units or product lines, not individual applications. If your flags are organized only by application, that mismatch can lead to confusion and make permission management harder than it needs to be.
Instead, organize flags in a way that matches how your teams actually work. People should be able to see the flags relevant to their work—whether that's by team, product, project, or application. They should also be able to inspect a flag's configuration and stay up to date on any changes.
Choose the right level of abstraction for grouping your flags:
- If your teams own features end-to-end, group flags by team.
- If you form cross-functional project teams, group flags by project.
Avoid putting all flags into one large group. While it might seem simpler, it quickly becomes difficult to manage. Permissions become complex and users are overwhelmed with irrelevant updates, making it harder to focus on what matters. Only include people in a flag group if they actually need to work with or track that flag.
### Make flags searchable globally
Modern applications are composed of multiple services with many complex dependencies. While organizing flags by applications, teams, or business units is a good starting point, real-world services often exist across these organizational boundaries.
Just because a user doesn't normally work with a flag doesn't mean they won't need to find it. However, you don't need to show every flag by default—especially those outside a user's main scope. What matters is that all flags are [searchable](https://docs.getunleash.io/reference/api/unleash/search-features). When someone searches for a flag, they should be able to view its configuration and ownership, so they can request access or submit a [change request](../../reference/change-requests) if needed.
This is why feature flag systems should be [open by default](./feature-flag-best-practices#5-choose-open-by-default).
There are also valid use cases for excluding flags from global search. For example, during sensitive projects, it may be necessary to hide flags that could expose confidential changes. Your system should support private flags in special cases—but those should remain the exception, not the norm.
### Design for flag permissions to change over time
The owner or maintainer of a flag may change over time as the feature it controls [evolves across its lifecycle](#use-feature-flag-lifecycle-to-optimize-your-development-workflow).
For instance, a developer might initially create a flag to hide incomplete code while working in [trunk-based development](https://www.getunleash.io/blog/how-to-implement-trunk-based-development-a-practical-guide#What%20is%20trunk-based%20development) model. At this stage, the flag exists to hide unfinished code and is owned and managed by the developer.
Later, when the feature is ready to roll out, the purpose of the flag shifts—from hiding unfinished work to managing exposure to users. With this shift, ownership often becomes shared:
- For simple rollouts, the developer might remain the main owner, gradually enabling the feature while monitoring for issues.
- For more complex rollouts, especially when business context matters, the developer may still manage the flag, but a product owner or Customer Success Manager might control targeting rules or decide which users see the feature first.
In B2B environments, for instance, it's common for customer-facing teams to guide rollouts based on account needs. These teams need the appropriate access and ownership—while still keeping developers in the loop for technical oversight.
Feature flag permissions should be dynamic and reflect real-world collaboration. You should also design your permission system to easily adapt to common scenarios like:
- Access is misconfigured—someone needs access or has access they shouldn't.
- Project requirements shift midway through development.
- Project ownership changes due to role or team changes.
- Organizational structure evolves and teams are reorganized.
### Get flag permissions right—and audit everything
As features move through their lifecycle, different teams will need access to the same flags—but not everyone should have the same level of control.
Here's a sample list of actions users may need to perform on a feature flag, each requiring its own permission setting:
- Create
- Delete
- Turn on / off
- Configure targeting
- Change rollout
- Read configuration
- Update configuration
Let's walk through a simple example: a product manager is collaborating with a developer to test a beta feature.
The developer should likely have full control—turning the flag on or off as needed. But what about the product manager? That depends on how your organization handles ownership and responsibilities. Your feature flag system should support fine-grained permissions to reflect those organizational needs.
Take targeting and rollout strategies, for instance:
- You might allow the product manager to update targeting rules but not control rollout percentages.
- Or, they may have full access to both, but changes must go through an approval workflow before being applied.
These choices often reflect broader compliance or security policies. The key is flexibility: your flag system should let you tailor permissions based on roles, use cases, and risk levels.
Just as important as getting permissions right is tracking what happens to your flags. A robust audit log is critical. It helps answer questions like:
- Who changed a targeting rule—and when?
- What configuration was live during an incident?
- Are approval workflows being followed?
- Do we need tighter controls or better communication?
And in the rare case of malicious behavior, an audit trail can help you pinpoint the issue and respond quickly.
### Align flag permissions with your global access controls
Your organization likely already defines user groups, for example by role (such as developers, operations, marketing) or by business unit. Use those existing structures to manage feature flag permissions. This makes it easier to assign the right access levels and keeps your permission model consistent across systems.
Your flag system should integrate with your [single sign-on](../../reference/sso) (SSO) provider. That way, when a user is added to or removed from a group in your central directory, their access to the [flag system updates automatically](../../reference/scim).
Managing access is complex—so don't manage it in multiple places. Let your global permissions drive flag access and keep everything in sync.
### Implement flag approval workflows early
Depending on the industry and legal framework you're operating in, you'll need varying levels of approval for a feature flag change to go into production. Whether your process is lightweight or highly controlled, your feature flag system should support it.
![Change request process](/img/change-request-flow.png)
In regulated industries, peer review is often required before changes reach production—commonly known as the [four-eyes principle](https://www.unido.org/overview-member-states-change-management-faq/what-four-eyes-principle). Think of it like a pull request: users should be able to bundle related changes across multiple flags into a single request, preview the impact, and submit it for review. Reviewers should have clear visibility into what's changing and what the new configuration will look like.
Even if your organization doesn't require approvals, having an optional review step is still valuable, as it allows you to:
- **Coordinate multiple changes**:
Grouping related flag updates ensures they go live together, preventing inconsistent or incomplete rollouts.
- **Prevent accidental changes**:
A quick review adds a layer of safety, so unfinished code won't accidentally be exposed.
- **Double-check outcomes**:
Reviewing changes in context helps you confirm that you're delivering what you intended.
## Staying in control with feature flags at scale
These best practices are based on insights from working with some of the world's largest organizations. What we've learned is that careful planning, discipline, and the right systems in place help you stay in control, no matter the size of your organization.
By [aligning feature flags with your organizational structure](#align-feature-flags-with-your-organizational-and-application-structure), and keeping them globally [searchable](../../reference/search-operators), you lay the foundation for a system that's [easy to maintain and scale](https://www.getunleash.io/unleash-enterprise-edge). [Instrumenting your code properly](#organize-your-feature-flag-code-effectively), [addressing technical debt regularly](#stay-on-top-of-technical-debt), and [maintaining a flexible permission system](#enable-secure-collaboration-with-flexible-permissions) ensures strong performance and security over time. And by avoiding common pitfalls—like embedding business logic or managing config through flags—you keep your system clean and manageable.
Feature flags are a powerful tool for delivering software efficiently while maintaining [security](https://www.getunleash.io/security-and-performance) and [compliance](../../using-unleash/compliance/compliance-overview). For developers working in large organizations, that is the best of both worlds. Following the practices in this guide will help you get the most value from feature flags without getting lost in complexity.
The key to success at scale? Clear processes, regular maintenance, and a commitment to best practices. With these in place, you can rely on feature flags to drive innovation and improve your software development lifecycle.
This guide focused on using [feature flags at scale](https://www.getunleash.io/blog/feature-management). For advice on building and scaling the flag system itself, check out our [feature flag best practices guide](./feature-flag-best-practices).

77
website/docs/topics/feature-flags/feature-flag-best-practices.mdx
View File

@ -1,21 +1,22 @@
---
title: "11 principles for building and scaling feature flag systems"
description: Build a scalable, secure feature flag system with 11 key principles. Improve DevOps metrics, ensure reliability, and enhance developer experience."
toc_max_heading_level: 2
---
import VideoContent from '@site/src/components/VideoContent.jsx';
Feature flags, sometimes called feature toggles or feature switches, are a powerful software development technique that allows engineering teams to decouple the release of new functionality from software deployments.
Feature flags, [sometimes called feature toggles or feature switches](../../what-is-a-feature-flag), are a powerful software development technique that allows engineering teams to decouple the release of new functionality from software deployments.
With feature flags, developers can turn specific features or code segments on or off at runtime without needing a code deployment or rollback. Organizations that adopt feature flags see improvements in key [DevOps metrics](https://www.getunleash.io/blog/dora-metrics-in-2023-5-ways-to-measure-devops-performance) like lead time to changes, mean time to recovery, deployment frequency, and change failure rate.
With feature flags, developers can turn [specific features or code segments on or off at runtime](https://www.getunleash.io/feature-flag-use-cases-software-kill-switches) without needing a code deployment or rollback. Organizations that adopt feature flags see improvements in key [DevOps metrics](https://www.getunleash.io/blog/dora-metrics-in-2023-5-ways-to-measure-devops-performance) like lead time to changes, mean time to recovery, deployment frequency, and change failure rate.
At Unleash, we've defined 11 principles for building a large-scale feature flag system. These principles have their roots in distributed systems design and focus on security, privacy, and scalability—critical needs for enterprise systems. By following these principles, you can create a feature flag system that's reliable, easy to maintain, and capable of handling heavy loads.
These principles are:
1. [Enable runtime control](#1-enable-runtime-control)
2. [Prioritize availability over consistency](#2-prioritize-availability-over-consistency)
3. [Make flags short-lived](#3-make-flags-short-lived)
2. [Make flags short-lived](#2-make-flags-short-lived)
3. [Prioritize availability over consistency](#3-prioritize-availability-over-consistency)
4. [Ensure unique flag names](#4-ensure-unique-flag-names)
5. [Choose open by default](#5-choose-open-by-default)
6. [Protect PII by evaluating flags server-side](#6-protect-pii-by-evaluating-flags-server-side)
@ -31,36 +32,25 @@ Let's dive deeper into each principle.
A scalable feature management system evaluates flags at runtime. Flags are dynamic, not static. If you need to restart your application to turn on a flag, that's configuration, not a feature flag.
A large-scale feature flag system that enables [runtime control](https://www.getunleash.io/blog/so-what-exactly-is-runtime-control) should have, at minimum, the following components: a service to manage feature flags, a database or data store, an API layer, a feature flag SDK, and a continuous update mechanism.
A large-scale feature flag system that enables [runtime control](https://www.getunleash.io/blog/so-what-exactly-is-runtime-control) should have, at minimum, the following components: a [service to manage feature flags](https://www.getunleash.io/blog/feature-management), a database or data store, an [API layer](../../understanding-unleash/unleash-overview), a [feature flag SDK](../../reference/sdks), and a continuous update mechanism.
Let's break down these components.
- **Feature Flag Control Service**: A service that acts as the control plane for your feature flags, managing all flag configurations. The scope of this service should reflect the boundaries of your organization.
### Feature flag system components
- **Feature Flag Control Service**: A [service that acts as the control plane for your feature flags](https://www.getunleash.io/feature-flag-service), managing all flag configurations. The scope of this service should reflect the boundaries of your organization.
- **Database or data store**: A robust, scalable, and highly available database or data store that stores feature flag configurations reliably. Common options include SQL databases, NoSQL databases, or key-value stores.
- **API layer**: An API layer that exposes endpoints for your application to interact with the _Feature Flag Control Service_. This API should allow your application to request feature flag configurations.
- **Feature flag SDK**: An easy-to-use interface for fetching flag configurations and evaluating feature flags at runtime. When considering feature flags in your application, the call to the SDK should query the local cache, and the SDK should ask the central service for updates in the background.
- **Continuous update mechanism**: An update mechanism that enables dynamic updates to feature flag configurations without requiring application restarts or redeployments. The SDK should handle subscriptions or polling to the _Feature Flag Control Service_ for updates.
- **API layer**: An API layer that exposes endpoints for your application to interact with the _Feature Flag Control Service_. [This API should allow your application to request feature flag configurations](../../understanding-unleash/unleash-overview).
- **Feature flag SDK**: [An easy-to-use interface for fetching flag configurations and evaluating feature flags at runtime](../../reference/sdks). When considering feature flags in your application, the call to the SDK should query the local cache, and the SDK should ask the central service for updates in the background.
- **Continuous update mechanism**: An update mechanism that enables [dynamic updates to feature flag configurations](../../understanding-unleash/unleash-overview) without requiring application restarts or redeployments. The SDK should handle subscriptions or polling to the _Feature Flag Control Service_ for updates.
![The SDK holds an in-memory feature flag configuration cache which is continuously synced with the Feature Flag Control Service. You can then use the SDK to check the state of feature flags in your application.](/img/feature-flag-scalable-architecture.png)
## 2. Prioritize availability over consistency
Your application shouldn't have any dependency on the availability of your feature flag system. Robust feature flag systems avoid relying on real-time flag evaluations because the unavailability of the feature flag system will cause application downtime, outages, degraded performance, or even a complete failure of your application.
If the feature flag system fails, your application should continue running smoothly. Feature flagging should degrade gracefully, preventing any unexpected behavior or disruptions for users.
You can implement the following strategies to achieve a resilient architecture:
- **Bootstrap SDKs with data**: Feature flagging SDKs should work with locally cached data, even when the network connection to the _Feature Flag Control Service_ is unavailable, using the last known configuration or defaults to ensure uninterrupted functionality.
- **Use local cache**: Maintaining a local cache of feature flag configurations helps reduce network round trips and dependency on external services. The local cache can periodically synchronize with the central _Feature Flag Control Service_ when it's available.
- **Evaluate feature flags locally**: Whenever possible, the SDKs or application components should evaluate feature flags locally without relying on external services, ensuring uninterrupted feature flag evaluations even if the feature flagging service is down.
- **Prioritize availability over consistency**: In line with the [CAP theorem](https://www.ibm.com/topics/cap-theorem), design for availability over strict consistency. In the face of network partitions or downtime of external services, your application should favor maintaining its availability rather than enforcing perfectly consistent feature flag configuration caches. Eventually consistent systems can tolerate temporary inconsistencies in flag evaluations without compromising availability.
## 3. Make flags short-lived
## 2. Make flags short-lived
The most common use case for feature flags is to manage the [rollout](https://www.getunleash.io/feature-flag-use-cases-progressive-or-gradual-rollouts) of new functionality. Once a rollout is complete, you should remove the feature flag from your code and archive it. Remove any old code paths that the new functionality replaces.
Avoid using feature flags for static application configuration. Application configuration should be consistent, long-lived, and loaded during application startup. In contrast, feature flags should be short-lived, dynamic, and updated at runtime. They prioritize availability over consistency and can be modified frequently.
Avoid using feature flags for static application configuration. Application configuration should be consistent, long-lived, and loaded during application startup. In contrast, [feature flags should be short-lived, dynamic, and updated at runtime](./best-practices-using-feature-flags-at-scale). They prioritize availability over consistency and can be modified frequently.
<table>
<thead>
@ -88,8 +78,8 @@ Avoid using feature flags for static application configuration. Application conf
</td>
<td>
<ul>
<li>A/B or multivariate testing</li>
<li>Gradual rollout</li>
<li>[A/B or multivariate testing](https://www.getunleash.io/blog/ab-testing-feature-flags-how-to)</li>
<li>[Gradual rollout](https://www.getunleash.io/feature-flag-use-cases-progressive-or-gradual-rollouts)</li>
<li>Permission flags (i.e., beta testing programs) </li>
<li>Operational testing in production</li>
</ul>
@ -98,22 +88,38 @@ Avoid using feature flags for static application configuration. Application conf
</tbody>
</table>
### Strategies for large organizations implementing feature flags
To succeed with feature flags in a large organization, [follow these strategies](https://docs.getunleash.io/topics/feature-flags/best-practices-using-feature-flags-at-scale):
- **Set flag expiration dates**: Assign expiration dates to feature flags to track which flags are no longer needed. A good feature flag management tool will alert you to expired flags, making it easier to maintain your codebase.
- **Treat feature flags like technical debt**: Incorporate tasks to remove outdated feature flags into your sprint or project planning, just as you would with technical debt. Feature flags add complexity to your code by introducing multiple code paths that need context and maintenance. If you don't clean up feature flags in a timely manner, you risk losing the context as time passes or personnel changes, making them harder to manage or remove.
- **Treat feature flags like technical debt**: Incorporate tasks to remove outdated feature flags into your sprint or project planning, just as you would with [technical debt](../../reference/technical-debt). Feature flags add complexity to your code by introducing multiple code paths that need context and maintenance. If you don't clean up feature flags in a timely manner, you risk losing the context as time passes or personnel changes, making them harder to manage or remove.
- **Archive old flags**: When feature flags are no longer in use, archive them after removing them from the codebase. This archive serves as an important audit log of feature flags and allows you to revive flags if you need to restore an older version of your application.
While most feature flags should be short-lived, there are valid exceptions for long-lived flags, including:
- **Kill switches**: [Kill switches](https://www.getunleash.io/blog/kill-switches-best-practice) act as inverted feature flags, allowing you to gracefully disable parts of a system with known weak spots.
- **Internal flags**: Internal flags to enable additional debugging, tracing, and metrics at runtime, which are too costly to run continuously. Engineers can enable these flags while debugging issues.
## 3. Prioritize availability over consistency
Your application shouldn't have any dependency on the availability of your feature flag system. Robust feature flag systems avoid relying on real-time flag evaluations because the unavailability of the feature flag system will cause application downtime, outages, degraded performance, or even a complete failure of your application.
If the feature flag system fails, your application should continue running smoothly. Feature flagging should degrade gracefully, preventing any unexpected behavior or disruptions for users.
You can implement the following strategies to achieve a resilient architecture:
- **Bootstrap SDKs with data**: Feature flagging SDKs should work with locally cached data, even when the network connection to the _Feature Flag Control Service_ is unavailable, using the last known configuration or defaults to ensure uninterrupted functionality.
- **Use local cache**: Maintaining a local cache of feature flag configurations helps reduce network round trips and dependency on external services. The local cache can periodically synchronize with the central _Feature Flag Control Service_ when it's available.
- **Evaluate feature flags locally**: Whenever possible, the SDKs or application components should evaluate feature flags locally without relying on external services, ensuring uninterrupted feature flag evaluations even if the feature flagging service is down.
- **Prioritize availability over consistency**: In line with the [CAP theorem](https://www.ibm.com/topics/cap-theorem), design for availability over strict consistency. In the face of network partitions or downtime of external services, your application should favor maintaining its availability rather than enforcing perfectly consistent feature flag configuration caches. Eventually consistent systems can tolerate temporary inconsistencies in flag evaluations without compromising availability.
## 4. Ensure unique flag names
Ensure that all flags within the same _Feature Flag Control Service_ have unique names across your entire system. [Unique naming](https://docs.getunleash.io/topics/feature-flags/best-practices-using-feature-flags-at-scale#13-never-reuse-feature-flag-names) prevents the reuse of old flag names, reducing the risk of accidentally re-enabling outdated features with the same name.
![Creating a feature flag with a unique name referencing the project and related jira ticket.](/img/feature-flag-unique-name.png)
### The benefits of unique naming
Unique naming has the following advantages:
- **Flexibility over time**: Large enterprise systems are not static. Monoliths may split into microservices, microservices may merge, and applications change responsibility. Unique flag naming across your organization means that you can reorganize your flags to match the changing needs of your organization.
- **Fewer conflicts**: If two applications use the same feature flag name, it can become difficult to identify which flag controls which application. Even with separate namespaces, you risk toggling the wrong flag, leading to unexpected consequences.
@ -124,10 +130,13 @@ Unique naming has the following advantages:
Making feature flag systems open by default enables engineers, product owners, and support teams to collaborate effectively and make informed decisions. Open access encourages productive discussions about feature releases, experiments, and their impact on the user experience.
Access control and visibility are also key considerations for security and compliance. Tracking and auditing feature flag changes help maintain data integrity and meet regulatory requirements. While open access is key, it's equally important to integrate with corporate access controls, such as SSO, to ensure security. In some cases, additional controls like feature flag approvals using the [four-eyes principle](/reference/change-requests) are necessary for critical changes.
[Access control and visibility](../../feature-flag-tutorials/use-cases/user-management-access-controls-auditing) are also key considerations for [security and compliance](https://www.getunleash.io/security-and-performance). Tracking and auditing feature flag changes help maintain data integrity and meet regulatory requirements. While open access is key, it's equally important to integrate with corporate access controls, such as SSO, to ensure security. In some cases, additional controls like feature flag approvals using the [four-eyes principle](/reference/change-requests) are necessary for critical changes.
![Preview of a change request to enable gradual rollout in production.](/img/feature-flag-change-requests-preview.png)
### Open collaboration strategies for feature flag management
For open collaboration, consider providing the following:
- **Access to the codebase**: Engineers need direct access to the codebase that contains the feature flags. This allows them to quickly diagnose and fix issues, minimizing downtime and performance problems.
- **Access to configuration**: Engineers, product owners, and even technical support should be able to view feature flag configuration. This transparency provides insights into which features are currently active, what conditions trigger them, and how they impact the application's behavior. Product owners can also make real-time decisions on feature rollouts or adjustments without relying solely on engineering resources.
@ -145,6 +154,8 @@ Let's look at an example where feature flag evaluation happens inside the server
Client-side applications where the code resides on the user's machine in browsers or mobile devices, require a different approach. You can't evaluate flags on the client side because it raises significant security concerns by exposing potentially sensitive information such as API keys, flag data, and flag configurations. Placing these critical elements on the client side increases the risk of unauthorized access, tampering, or data breaches.
### Evaluate within a self-hosted environment
Instead of performing client-side evaluation, a more secure and maintainable approach is to evaluate feature flags within a self-hosted environment. Doing so can safeguard sensitive elements like API keys and flag configurations from potential client-side exposure. This strategy involves a server-side evaluation of feature flags, where the server makes decisions based on user and application parameters and then securely passes down the evaluated results to the frontend without any configuration leaking.
![In client-side setups, perform the feature flag evaluation on the server side. Connected client-side applications receive only evaluated feature flags to avoid leaking configuration.](/img/feature-flag-architecture-client-side.png)
@ -181,6 +192,8 @@ When designing a scalable feature flag system, one of the most effective strateg
By decoupling read and write operations, you gain the flexibility to scale horizontally based on the unique demands of your application. For example, if read traffic increases, you can add more servers or containers to handle the load without needing to scale the write operations.
### The benefits of decoupling
The benefits of decoupling read and write operations extend beyond just scalability; let's look at a few others:
- **More efficient caching**: You can optimize your flag caching for read operations to reduce latency while keeping write operations consistent.
- **Granular access control**: You can apply different security measures and access controls to the two APIs, reducing the risk of accidental or unauthorized changes.
@ -194,6 +207,8 @@ Minimizing the size of feature flag payloads is a critical aspect of maintaining
Avoid storing large user lists directly in the feature flag configuration, which can lead to scaling issues. Instead, categorize users into logical groups at a higher layer (for example, by subscription plan or location) and use group identifiers for targeting within the feature flag system.
### The benefits of small payloads
Keeping the feature flag payload small results in:
- **Reduced network load**: Large payloads can lead to increased network traffic between the application and the feature flagging service. This can overwhelm the network and cause bottlenecks, leading to slow response times and degraded system performance. Even small optimizations can make a big difference at scale.
@ -210,7 +225,7 @@ For more insights into reducing payload size, visit our [Best practices for usin
Feature flagging solutions are indispensable tools in modern software development, enabling teams to manage feature releases and experiment with new functionality. However, one aspect that is absolutely non-negotiable in any feature flag solution is the need to ensure a consistent user experience. Feature flagging solutions must prioritize consistency and guarantee the same user experience every time, especially with percentage-based gradual rollouts.
Strategies for consistency in percentage-based gradual rollouts:
### Strategies for consistency in percentage-based gradual rollouts
- **User hashing**: Assign users to consistent groups using a secure hashing algorithm based on unique identifiers like user IDs or emails. This ensures that the same user consistently falls into the same group.
- **Segmentation control**: Provide controls within the feature flagging tool to allow developers to [segment](/reference/segments) users logically by criteria like location, subscription type, or other relevant factors, ensuring similar experiences for users within the same segment.
@ -222,6 +237,8 @@ Strategies for consistency in percentage-based gradual rollouts:
[Developer experience](https://www.opslevel.com/resources/devex-series-part-1-what-is-devex) is a critical factor to consider when implementing a feature flag solution. A positive developer experience enhances the efficiency of the development process and contributes to the overall success and effectiveness of feature flagging. One key aspect of developer experience is ensuring the testability of the SDK and providing tools for developers to understand how and why feature flags are evaluated.
### Developer experience strategies for feature flags
To ensure a good developer experience, you should provide the following:
- **Simplified testing and debugging**: A testable SDK enables developers to quickly test and iterate on features, speeding up development cycles. Developers can toggle flags, simulate conditions, and observe results without significant code changes or redeployments. This makes it easier to identify and fix issues by examining flag configurations and logging decisions.
- **Visibility into flag behavior**: Developers need tools to understand how feature flags impact the user experience. Visibility into flag behavior helps them assess changes, debug effectively with multiple inputs, and collaborate more easily within cross-functional teams.
@ -233,7 +250,7 @@ To ensure a good developer experience, you should provide the following:
## Additional resources
Thank you for reading. Our motivation for writing these principles is to share what we've learned building a large-scale feature flag solution with other architects and engineers solving similar challenges. Unleash is open-source, and so are these principles.
Thank you for reading. Our motivation for writing these principles is to share what we've learned building a [large-scale feature flag solution](https://www.getunleash.io/enterprise-feature-management-platform) with other architects and engineers solving similar challenges. [Unleash is open-source](https://www.getunleash.io/open-source), and so are these principles.
Have something to contribute? [Open a PR](https://github.com/Unleash/unleash/pulls) or [discussion](https://github.com/orgs/Unleash/discussions) on our GitHub.
To learn more about Unleash architecture and implementation basics, check out this webinar:

2
website/docs/understanding-unleash/the-anatomy-of-unleash.mdx
View File

@ -120,7 +120,7 @@ Segments are only available to [Pro](/availability#plans) and [Enterprise](https
## Variants and feature flag payloads
By default, a [feature flag](../reference/feature-toggles) in Unleash only tells you whether a feature is enabled or disabled, but you can also add more information to your flags by using [**feature flag variants**](../reference/feature-toggle-variants). Variants also allow you to run [A/B testing experiments](../feature-flag-tutorials/use-cases/a-b-testing).
By default, a [feature flag](../reference/feature-toggles) in Unleash only tells you whether a feature is enabled or disabled, but you can also add more information to your flags by using [**feature flag variants**](../reference/feature-toggle-variants). Variants also allow you to run [A/B testing experiments](../../feature-flag-tutorials/use-cases/a-b-testing).
Feature flags are designed to let you decide which users get access to a feature. Variants are designed to let you decide **which version** of the feature a user gets access to. For instance, if user A is part of your beta testing program and gets access to a new beta feature, then you can use variants to decide whether they should get the red version or the green version of the feature.

BIN
website/static/img/evaluate-flags-at-the-edge.jpg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 80 KiB

21
website/static/img/evaluate-flags-at-the-edge.svg
View File

File diff suppressed because one or more lines are too long
Side by Side

Before

Width:  |  Height:  |  Size: 8.1 KiB

BIN
website/static/img/evaluate-flags-in-many-places.jpg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 149 KiB

BIN
website/static/img/evaluate-flags-only-once.jpg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 65 KiB

21
website/static/img/evaluate-flags-only-once.svg
View File

File diff suppressed because one or more lines are too long
Side by Side

Before

Width:  |  Height:  |  Size: 32 KiB

21
website/static/img/feature-flag-organization-methods.svg
View File

File diff suppressed because one or more lines are too long
Side by Side

Before

Width:  |  Height:  |  Size: 36 KiB