In the world of cloud infrastructure management, consistent and accurate resource tagging is crucial. It's nice to be able to detect when GCP labels deviate from your standards, but even better to correct those deviations automatically. Enter the GCP Labels mod for Flowpipe, a new tool that enables you to automate label management using a declarative ruleset. Let's take a look.
Add mandatory labels
Here's a Steampipe query that reveals incorrectly-labeled GCP buckets.
select name, labels from gcp_storage_bucket where name ~ 'flowpipe'+--------------------+------------------+| name | labels |+--------------------+------------------+| flowpipe-bucket-01 | {"cc":"default"} || flowpipe-bucket-02 | <null> |+--------------------+------------------+
First, we'd like to apply the mandatory label env:dev
to all of them. To do that, we'll use the detect_and_correct_storage_buckets_with_incorrect_labels
pipeline in the GCP Labels mod.
Per the README, the setup entails:
Cloning the mod
Visiting its directory and installing dependencies (
flowpipe mod install
)Starting Steampipe as a service (
steampipe service start
)Using
credential_import
to give Flowpipe access to the credentials in Steampipegcp.spc
Configuring label rules
We'll use these definitions in add-notify.fpvars
.
approvers = []base_label_rules = {add = {env = "dev"}}
With approvers = []
, the mod will report label violations to the default notifier, the terminal, but take no action. The only rule is to add our desired mandatory label.
Now we'll run the pipeline to detect and correct bucket labels, and point it at that package of variables.
$ flowpipe pipeline run detect_and_correct_storage_buckets_with_incorrect_labels --var-file=add-notify.fpvars
Flowpipe reports the changes it would make.
Detected flowpipe-bucket-01 (flowpipe-bucket-01/encoded-blend-333001/) with incorrect labels. Labels that will be added or updated: env=dev.Detected flowpipe-bucket-02 (flowpipe-bucket-02/encoded-blend-333001/) with incorrect labels. Labels that will be added or updated: env=dev.
To make the changes, we'll use add-wizard.fpvars
, with the same label rules but with approvers = ["default"]
. That activates an input step that will (in this case) prompt to Skip
or Apply
the proposed change.
Flowpipe pauses on the first detected violation.
$ flowpipe pipeline run detect_and_correct_storage_buckets_with_incorrect_labels --var-file=add-wizard.fpvars┃ Detected flowpipe-bucket-01 (flowpipe-bucket-01/encoded-blend-333001/) with incorrect labels. Labels that will be added or updated: env=dev.┃ > Skip┃ Apply
We'll down-arrow and choose Apply
. Flowpipe reports:
Applied changes to labels on flowpipe-bucket-01.
Because we specified the default approver, this interaction happens in the console. You could shift the interaction to Slack, MSTeams, or email by naming — in approvers
— one or more notifiers configured with alternate integrations.
We'll choose Apply
for the second bucket and recheck the situation.
+--------------------+------------------------------------------+| name | labels |+--------------------+------------------------------------------+| flowpipe-bucket-01 | {"cc":"default","env":"dev"} || flowpipe-bucket-02 | {"env":"dev"} |+--------------------+------------------------------------------+
That's a good start, but we now want to correct cc
, that's a bogus label that some people use for cost_center
. Let's fix that.
Update label keys
We'll use this ruleset in update-fix.fpvars
.
approvers = []base_label_rules = {update_keys = {cost_center = ["~*:^cc|cost_centre$"]}}incorrect_labels_default_action = "apply"
With approvers = []
the rules run without pausing for input.
The update_keys
rule uses pattern matching to converge a set of bogus labels (cc
or cost_centre
) onto cost_center
.
The default incorrect_labels_default_action
is notify
, which would be a dry run, but we want to apply the fix so we set it to apply
.
We run the pipeline again, with this new package of variables.
$ flowpipe pipeline run detect_and_correct_storage_buckets_with_incorrect_labels --var-file=update-fix.fpvars
Flowpipe reports:
Applied changes to labels on flowpipe-bucket-01.
And here's the new situation.
+--------------------+---------------------------------------+| name | labels |+--------------------+---------------------------------------+| flowpipe-bucket-01 | {"cost_center":"default","env":"dev"} || flowpipe-bucket-02 | {"env":"dev"} |+--------------------+---------------------------------------+
Label operations
We've seen add
and update_keys
, here's the full set of operations documented in the README.
Add. Ensure resources have mandatory labels.
Remove. Remove specified labels.
Remove Except. Inverse of remove: remove all but specified labels.
Update Keys. Rewrite label keys.
Update Values. Rewrite label values.
To match labels you can use literal strings, or Postgres regular expressions.
Automate the fixes
It would be tedious to apply these kinds of changes with one-at-a-time approval. Let's automate with a new ruleset in continuous-compliance.fpvars
.
approvers = []base_label_rules = {update_keys = {cost_center = ["~*:^cc|cost_centre$"]}add = {env = "dev"}}storage_buckets_with_incorrect_labels_trigger_enabled = truestorage_buckets_with_incorrect_labels_trigger_schedule = "daily"incorrect_labels_default_action = "apply"
Note that you can combine operations like add
and update_keys
to build up a complex ruleset.
Here we're also using config variables to active the pipeline's query trigger, define its schedule, and define the default action (apply
). With these settings, Flowpipe will run the pipeline daily apply the changes we've seen to new buckets.
To run the pipeline on a schedule, use Flowpipe in server mode.
flowpipe server --var-file=continuous-compliance.fpvars
Nothing will happen the first day because we've already fixed the buckets. But if we create flowpipe-bucket-03
with no labels it will — within a day — automagically be labeled with env:dev
.
Continuous compliance for labels
The GCP Labels mod for Flowpipe enables flexible and powerful management of GCP labels at scale. With declarative rulesets and automated workflows, you can ensure consistent labeling across your GCP infrastructure with or without manual intervention. Whether you're looking to standardize existing labels, enforce new labeling policies, or maintain ongoing compliance, this mod provides the tools you need to streamline your labeling discipline. Give it a try and please let us know how it goes.