Bug fixes

• trigger run command now exits when the execution is paused. (#962).
• Event jsonl output file deletion is now handled correctly. (#960).

Bug fixes

• Added missing checks to the 1.12, 1.16, and 1.22 pipelines in CIS v3.0.0 and v4.0.0. (#3)

Bug fixes

• Fix crashing cases when using --output json. (#594).
• Coerce variables set in interactive console to their declared type. (#595).
• Nested pipelines now correctly pauses parent pipelines. (#955).
• Pipeline with max_concurrency setting is now automatically paused and will successfully resume. (#957).
• form_url is now sanitized.

What's new?

• Added 84 new 'detect and correct' pipelines to identify AWS resources that are non-compliant with common security and compliance checks. These pipelines can also remediate non-compliant automatically or with approval steps. For usage information and a full list of pipelines, please see AWS Compliance Mod.

What's new?

• Added pipelines to run CIS v3.0.0 and v4.0.0 benchmarks. These pipelines can be used to identify AWS resources that are non-compliant with CIS recommendations and also remediate them according to CIS remediation suggestions. For usage information and a full list of pipelines, please see AWS CIS Mod.

We're excited to announce the v1.0.0 release of all 76 Flowpipe mods, including 29 Library mods, 6 Standard mods, and 41 Sample mods!

Breaking changes

• Flowpipe v1.0.0 is now required. For a full list of CLI changes, please see the Flowpipe v1.0.0 CHANGELOG.
• In Flowpipe configuration files (.fpc), credential and credential_import resources have been renamed to connection and connection_import respectively.
• Updated the following param types:
  • approvers: list(string) to list(notifier).
  • database: string to connection.steampipe.
  • notifier: string to notifier.
• Updated the following variable types:
  • approvers: list(string) to list(notifier).
  • database: string to connection.steampipe.
  • notifier: string to notifier.
• Renamed cred param to conn and updated its type from string to conn.

What's new?

• connection resource to manage credentials. Documentation.
• connection and notifier types for variables and params. (#871)
• enum validation for variables and params.
• Defined exit codes for various CLI operations. Documentation.

Bug fixes

• Passing pipeline references to nested mods for execution. (#908)
• Do not crash if pipeline reference is set to a string. (#911)

Deprecation

• credential and credential_import are deprecated to be replaced with connection and connection_import.

Enhancements

• Added an optional param blocks to the post_message pipeline. (#24) (Thanks @johnlayton for the contribution!)

Bug fixes

• Fixed put_s3_bucket_encryption pipeline to correctly encrypt an S3 bucket without returning an error. (#68) (Thanks @gcasilva for the contribution!)

Bug fixes

• trigger introspection output correctly shows param attribute. (#900)

What's new?

• tags argument in pipeline param and mod variable resources. (#898).
• Updated Docker dependency to v27.1.2.

Bug fixes

• source attribute in function step is now evaluated relative to the its mod directory rather than the root mod directory. (#895).

What's new?

• trigger list command includes triggers from root mod's immediate dependencies. (#892).

Bug fixes

• Function step will no longer randomly fail in slower host machines. (#888).
• Mod variable definition now matches Powerpipe's definition. (#889).

What's new?

• Added detect_and_correct_sql_database_instances_with_incorrect_labels pipleine for SQL Database Instance. (#6)

Enhancements

• Added a default value for the base_label_rules variable. (#7)

Enhancements

• Added a default value for the base_tag_rules variable. (#18)

Enhancements

• Added a default value for the base_tag_rules variable. (#28)

Bug fixes

• Complex nested map data type in pipeline param no longer fails with a mismatched types error. (#879).

What's new?

• On-demand trigger execution. (#864).
• param support for trigger. (#840).

Bug fixes

• Complex data type in pipeline param no longer fails with a mismatched types error. (#879).
• Pipeline param default value is not nested in a map data type. (#880).

Bug fixes

• The variable command no longer fails if the .flowpipe directory in the user's home directory is not created yet. (#872).

What's new?

• Interactive workflows in the terminal via console integration. Blog.
• Simplified progress output for flowpipe pipeline run command when running in Client mode and not using the --verbose arg.
• --data-dir parameter to specify the location of the event store database. (#852).
• --execution-id parameter to specify custom execution id for pipeline run. (#856).
• Update Go version to v1.22.4.

Bug fixes

• Return a non-zero exit code if there's a failure. (#855).
• loop block now respect the if step attribute. (#858).

What's new?

• Added 22 detect and correct pipelines to identify unused and underutilized GCP resources, as well as deprecated resource configurations. These pipelines also suggest potential remediation actions to optimize costs. For usage information and a full list of pipelines, please see GCP Thrifty Mod.

What's new?

• Detect and correct misconfigured labels across 8 GCP resource types.
• Automatically add mandatory labels (e.g. env, owner).
• Clean up prohibited labels (e.g. secret, key).
• Reconcile shorthand or misspelled label keys to standardized keys (e.g. cc to cost_center).
• Update label values to conform to expected standards, ensuring consistency (e.g. Prod to prod).

For detailed usage information and a full list of pipelines, please see GCP Labels Mod.

What's new?

• Added 24 detect and correct pipelines to identify unused and underutilized Azure resources, as well as deprecated resource configurations. These pipelines also suggest potential remediation actions to optimize costs. For usage information and a full list of pipelines, please see Azure Thrifty Mod.

What's new?

• Detect and correct misconfigured tags across 55+ Azure resource types.
• Automatically add mandatory tags (e.g. env, owner).
• Clean up prohibited tags (e.g. secret, key).
• Reconcile shorthand or misspelled tag keys to standardized keys (e.g. cc to cost_center).
• Update tag values to conform to expected standards, ensuring consistency (e.g. Prod to prod).

For detailed usage information and a full list of pipelines, please see Azure Tags Mod.

What's new?

• The mod has been updated to run in the Wizard mode by default.

What's new?

• Detect and correct misconfigured tags across 65+ AWS resource types.
• Automatically add mandatory tags (e.g. env, owner).
• Clean up prohibited tags (e.g. secret, key).
• Reconcile shorthand or misspelled tag keys to standardized keys (e.g. cc to cost_center).
• Update tag values to conform to expected standards, ensuring consistency (e.g. Prod to prod).

For detailed usage information and a full list of pipelines, please see AWS Tags Mod.

What's new?

• Add support for installing mods from a branch or from the local file system. (#849).

  To install from a branch:

  flowpipe mod install github.com/turbot/flowpipe-mod-aws-thrifty#main

  To reference a mod in the local file system:

  flowpipe mod install ../mods/local_mod_folder

• Add --pull flag to mod command to control the mod update strategy. (#849). Possible update strategies are:

  • full - check branch and tags for both latest and accuracy
  • latest - update everything to latest, but only branches - not tags - are commit checked (which is the same as latest)
  • development - update branches and broken constraints to latest, leave satisfied constraints unchanged
  • minimal - only update broken constraints, do not check branches for new commits

• Variable list and show commands. (#373)

Bug fixes

• Pipeline references declared in subsequent files are correctly identified and processed.
• Preserves pipeline params ordering as specified in the pipeline definition. (#408)

What's new?

• Added Detect and Correct pipeline for DynamoDB tables with stale data. (#34)

What's new?

• Added the following new pipeline:
  • delete_dynamodb_table

Bug fixes

• Load locals in order of dependency. (#399).

What's new?

• Added 30 new 'detect and correct' pipelines to identify unused and underutilized AWS resources, as well as deprecated resource configurations. These pipelines also suggest potential remediation actions to optimize costs. For usage information and a full list of pipelines, please see AWS Thrifty Mod.

Bug fixes

• Pipeline execution no longer stalls when concurrency limit is applied and if clause returns false. (#836).
• Trigger's common attributes (title, description, tags, documentation) allow functions and expresions. (#394).

Bug fixes

• Param can be used in query step's args attribute. (#830).
• File watcher now correctly detect changes in the loop block. (#808).
• Duplicate step names are now detected and reported as an error. (#820).
• Better error message for invalid notifier reference. (#826).

Bug fixes

• Lazy create flowpipe.db. (#808).
• Respect max_concurrency in pipeline and input steps. (#815).
• Misleading error message for invalid step dependencies. (#816).
• HTTP integration address is shown correctly at the beginning of each input step loop. (#818).

Bug fixes

• loop block now works in container, function, message and input steps.
• Use HCL expressions in max_concurrency step argument. (#800).
• throw, retry and error block now works for input step.

Bug fixes

• Input step respects the max_concurrency argument. (#798).
• Erroneous error message detecting a missing credential where there isn't one.
• HCL try() function should be evaluated at runtime rather than parse time.
• Integration and input step URLs should use the provided custom host & port. (#792).
• Shows filename and line number for invalid step references.

What's new?

• Microsoft Teams integration. Documentation.

Bug fixes

• Function step output attribute should be called response not result. (#789).
• Pipeline execution should not fail when a string argument is passed with double quotes. (#791).

Bug fixes

• Multiselect Inputs with preselected Options now correctly pre-populate in Slack.
• Change detection in throw and output block in pipeline steps works correctly with ternary operators and will not trigger mod reload for white space changes.

Bug fixes

• Multi-select option in input step now works. (#776).
• Input step white space changes will not trigger mod reload. (#297).

What's new?

• Workflow - message step for easy notifications. Documentation.
• Workflow - input step for buttons, text and other data. Documentation.
• Workflow - simple, reusable integration and notifier configuration for HTTP, Slack and Email. Documentation
• Import Steampipe connections as Flowpipe credentials. Documentation.
• Manage concurrency of pipelines and steps.
• New credential types: alicloud and mastodon.
• Shorter hash for HTTP triggers for simpler URLs.
• DuckDB support in query step & trigger.
• Step metadata, like started_at and finished_at added under a flowpipe attribute.
• Moved flowpipe.db into the mod-level .flowpipe directory.
• connection_string in query step and trigger renamed to database.

Deprecation

• Email step. Please use the message step instead.

Bug fixes

• log_level workspace setting is now respected (#618).
• Default listen flag should be network, not localhost (#694)
• Trigger attributes are now validated (#225).
• Pipeline output attributes are now validated (#239).
• Pipeline param default value data type is now validated against the specified type (#262).
• Removed titles when merging multiple error messages (#263).
• Runtime resolution of pipeline reference and credentials are now working correctly. (#732).
• Scheduled triggers are now re-scheduled when mod files have changed.
• File watcher reliability improvements.

Bug fixes

• Updated metadata param type in create_ticket pipeline to be consistent with similar param types.

Bug fixes

• Fixed secret param type in create_secret pipeline.

Bug fixes

• Fixed broken links to credential import docs in various sample READMEs.

What's new?

• Added the following new sample mods: (#108)
  • add_s3_bucket_cost_center_tags
  • aws_iam_access_key_events_notifier_with_multiple_pipelines
  • aws_iam_access_key_events_notifier_with_single_pipeline
  • deactivate_expired_aws_iam_access_keys_using_queries
  • deactivate_expired_aws_iam_access_keys_with_approval
  • notify_new_aws_iam_access_keys

Enhancements

• Updated all AWS, Azure, PagerDuty, Slack, Zendesk library mod dependency versions in several sample mods. (#108)

Bug fixes

• Removed duplicate ticket_id param from update_ticket_comment pipeline.

Bug fixes

• Fixed invalid type for license param in create_user pipeline. (#6)

Bug fixes

• Fixed mismatched types for generate_ssh_keys param in various Compute VM test pipelines.

Bug fixes

• Fixed the type mismatch of the input parameter in the get_channel_history pipeline. (#20)

Bug fixes

• Only trigger pipeline failure after a step has completed all retries (#630).
• DOCKER_HOST, DOCKER_API_VERSION, DOCKER_CERT_PATH, DOCKER_TLS_VERIFY environment variables are now correctly passed to the Docker client (#651).
• Do not set memory_swappiness when using Podman (#652).

What's new?

• Added create_branch, delete_branch and get_branch pipelines. (#10)

What's new?

• Added pipeline get_channel_id. (#17).

Bug fixes

• Fixed the input parameters of the test_post_message pipeline. (#17)

Bug fixes

• Build error no longer suppressed in container and function steps (#625).
• Handles complex data types in step output (#626).

Bug fixes

• Fix the commands in add_labels_to_compute_disk and add_labels_to_compute_instance pipelines. (#7)

Bug fixes

• Map MySQL query results to correct types (#604).
• Handle null values in query trigger results (#611).
• Convert binary data in query results to a string.
• Docker containers now clear the cache to get correct parameters (#561).
• Improved error message when Flowpipe CLI port is already in use (#603).

What's new?

• Query trigger type to watch & event on database changes. Documentation.
• HTTP trigger can now handle both GET and POST methods. Documentation.
• Query steps & triggers now support Postgres, MySQL, SQLite, and Postgres.
• Define container step using a source argument for inline image definitions.
• Add a timeout to pipeline steps.
• Enable or disable triggers using the enabled attribute.
• Improved and expanded output for flowpipe server.
• Improved and standardized output for CLI list and show commands.
• Expanded intervals available in schedule and query triggers (e.g. 5m, 10m, etc).
• New credential types: BitBucket, Datadog, Freshdesk, JumpCloud, ServiceNow, Turbot Guardrails.
• Automatic check & notify for new CLI versions.

Bug fixes

• Implemented a more descriptive error message for server startup failures.
• Fixed Step Arguments unable to be referenced in the Pipeline definition.
• Added missing execution_mode argument to HTTP Trigger (#533).
• Fixed args arguments unable to be updated in the Pipeline Step loop block (#559).
• Fixed an issue in the bootstrap process for identifying the config path.

What's new?

• Added the query_and_stop_aws_ec2_instances_by_tag sample mod that can be used with Flowpipe. (#99)

Bug fixes

• Fixed the README docs to use --arg instead of --pipeline-arg as the argument flag. (#102)
• Fixed the link for installing mod dependencies in all the README docs. (#98)

Bug fixes

• Updated the credential section of README to use api_key instead of token. (#7)

Bug fixes

• Removed inaccurate SQL Query string validation to check for arguments. (#516)

35 new, ready-to-use Flowpipe sample mods are now available! These mods serve as practical examples, showcasing the patterns and applications of various library mods. Every mod comes with specific instructions for installation and use, enabling fast and easy setup.

A full list of sample mods can be found in the Flowpipe Hub and the source code is available at turbot/flowpipe-samples.

Introducing Flowpipe, a cloud scripting engine. Automation and workflow to connect your clouds to the people, systems and data that matter. Pipelines for DevOps written in HCL.

Initial support for:

• Pipeline execution
• Steps: container, email, function, http, pipeline, query, sleep, transform
• Triggers: schedule, http
• Credential management
• Mod composition

Learn more at:

• Website - https://flowpipe.io
• Docs - https://flowpipe.io/docs
• Hub - https://hub.flowpipe.io
• Introduction - https://flowpipe.io/blog/introducing-flowpipe

We're thrilled to announce the release of 28 new Flowpipe library mods, featuring versatile pipelines for common tasks. These include starting AWS EC2 instances, creating GitHub issues, sending Slack messages, generating Zendesk tickets, and much more!

A full list of library mods can be found in the Flowpipe Hub.

For more information on how you can get started incorporating these library mods into your own mods and pipelines, please see Introducing Flowpipe - Composable Mods.