Terraform

The Terraform scraper reads Terraform state files and creates configuration items from the resources defined in the state. This enables you to track your infrastructure as code resources and monitor changes to your Terraform-managed infrastructure.

Use Cases

• Infrastructure Tracking: Monitor all resources managed by Terraform
• State Drift Detection: Track changes between Terraform state and actual infrastructure
• Multi-Environment Management: Track resources across different environments and stages
• Resource Dependency Mapping: Understand relationships between Terraform-managed resources
• Compliance Monitoring: Ensure infrastructure remains compliant with defined configurations

terraform-scraper.yaml
---
apiVersion: configs.flanksource.com/v1
kind: ScrapeConfig
metadata:
  name: terraform
spec:
  schedule: '@every 5m'
  terraform:
    - name: '{{ filepath.Base .path}}'
      state:
        s3:
          bucket: terraform
          connection: connection://aws
          objectPath: 'states/**/*.tfstate'

Field	Description	Scheme	Required
schedule	Specify the interval to scrape in cron format. Defaults to every 60 minutes.	Cron
retention	Settings for retaining changes, analysis and scraped items	Retention
terraform	Specifies the list of Terraform configurations to scrape.	[]Terraform	true

Terraform

Field	Description	Scheme
name*	Template for naming the configuration items created from Terraform resources	GoTemplate
state*	Configuration for accessing Terraform state files	TerraformStateSource
transform	Transformations to apply to scraped items	Transform

TerraformStateSource

Field	Description	Scheme
gcs	Google Cloud Storage backend configuration for accessing state files	GCSConnection
local	Local file path to Terraform state file	string
s3	S3 backend configuration for accessing state files stored in AWS S3	S3Connection

S3Connection

Field	Description	Scheme
bucket*	S3 bucket name containing the state files	string
objectPath*	Path pattern to state files (supports glob patterns like states/**/*.tfstate)	string
accessKey	AWS access key for authentication	EnvVar
connection	Connection reference for AWS credentials	string
region	AWS region where the bucket is located	string
secretKey	AWS secret key for authentication	EnvVar

GCSConnection

Field	Description	Scheme
bucket*	GCS bucket name containing the state files	string
objectPath*	Path pattern to state files in the bucket	string
credentials	GCP service account credentials JSON	EnvVar

Configuration Examples

S3 Backend

apiVersion: configs.flanksource.com/v1
kind: ScrapeConfig
metadata:
  name: terraform-s3-states
spec:
  schedule: '@every 5m'
  terraform:
    - name: '{{ filepath.Base .path }}'
      state:
        s3:
          bucket: terraform-state-bucket
          connection: connection://aws
          objectPath: 'environments/**/*.tfstate'

Multiple Environment Tracking

apiVersion: configs.flanksource.com/v1
kind: ScrapeConfig
metadata:
  name: terraform-multi-env
spec:
  terraform:
    - name: 'prod-{{ filepath.Base .path }}'
      state:
        s3:
          bucket: terraform-prod-states
          connection: connection://aws-prod
          objectPath: 'prod/**/*.tfstate'
    - name: 'staging-{{ filepath.Base .path }}'
      state:
        s3:
          bucket: terraform-staging-states
          connection: connection://aws-staging
          objectPath: 'staging/**/*.tfstate'
    - name: 'dev-{{ filepath.Base .path }}'
      state:
        s3:
          bucket: terraform-dev-states
          connection: connection://aws-dev
          objectPath: 'dev/**/*.tfstate'

GCS Backend

apiVersion: configs.flanksource.com/v1
kind: ScrapeConfig
metadata:
  name: terraform-gcs-states
spec:
  terraform:
    - name: 'gcp-{{ filepath.Base .path }}'
      state:
        gcs:
          bucket: terraform-state-gcs
          objectPath: 'projects/**/*.tfstate'
          credentials:
            valueFrom:
              secretKeyRef:
                name: gcp-credentials
                key: service-account.json

Local State Files

apiVersion: configs.flanksource.com/v1
kind: ScrapeConfig
metadata:
  name: terraform-local-state
spec:
  terraform:
    - name: 'local-terraform'
      state:
        local: '/path/to/terraform.tfstate'

Masking Sensitive Data

Terraform state files store all resource attributes in plain text JSON, including sensitive values like passwords, API keys, and connection strings. For example, an AWS RDS instance in the state file contains:

{
  "mode": "managed",
  "type": "aws_db_instance",
  "name": "example",
  "provider": "provider[\"registry.terraform.io/hashicorp/aws\"]",
  "instances": [
    {
      "attributes": {
        "address": "mydb.123456789012.us-east-1.rds.amazonaws.com",
        "engine": "mysql",
        "engine_version": "5.7",
        "master_username": "admin",
        "master_password": "MyPa$$w0rd123!",  // Stored in plain text!
        "endpoint": "mydb.123456789012.us-east-1.rds.amazonaws.com:3306"
      }
    }
  ]
}

Default Masking Behavior

info

The Terraform scraper automatically masks common sensitive patterns by default. You don't need to configure anything for basic secret protection.

The following patterns are automatically masked:

Pattern	Fields Masked	Example
Passwords	*password*, *passwd*, *pwd*	master_password, db_password, user_pwd
Keys	*private_key*, *secret_key*, *api_key*, *access_key*	aws_secret_access_key, private_key_pem
Tokens	*token*, *auth*	github_token, auth_token, bearer_token
Secrets	*secret*	client_secret, webhook_secret
Connection Strings	Fields containing connection URIs	connection_string, database_url

Custom Masking Examples

For additional masking beyond the defaults, you can add custom rules. Note that Mission Control uses specific resource type formats (e.g., AWS::RDS::DBInstance, AWS::IAM::User):

apiVersion: configs.flanksource.com/v1
kind: ScrapeConfig
metadata:
  name: terraform-security-masking
spec:
  terraform:
    - name: '{{ .path | filepath.Base }}'
      state:
        s3:
          bucket: terraform-states
          connection: connection://aws
          objectPath: '**/*.tfstate'
      transform:
        mask:
          # Mask RDS and Aurora passwords with hash for change detection
          - selector: |
              config.type == 'AWS::RDS::DBInstance' ||
              config.type == 'AWS::RDS::DBCluster' ||
              config.type == 'AWS::RDS::DBProxy'
            jsonpath: $.config.master_password
            value: md5sum

          # Mask DocumentDB passwords
          - selector: config.type == 'AWS::DocDB::DBCluster'
            jsonpath: $.config.master_password
            value: md5sum

          # Mask ElastiCache auth tokens
          - selector: config.type == 'AWS::ElastiCache::CacheCluster'
            jsonpath: $.config.auth_token
            value: '***REDACTED***'

          # Mask IAM user access keys
          - selector: config.type == 'AWS::IAM::User'
            jsonpath: $.config.access_key_id
            value: '***ACCESS_KEY***'

          # Mask Lambda environment variables
          - selector: config.type == 'AWS::Lambda::Function'
            jsonpath: $.config.environment.variables[?(@.key =~ /.*secret.*|.*password.*|.*key.*/i)]
            value: md5sum

          # Mask custom application secrets in EC2 user data
          - selector: config.type == 'AWS::EC2::Instance'
            jsonpath: $.config.user_data
            value: '***USER_DATA_REDACTED***'

Excluding Sensitive Resources

For highly sensitive resources that shouldn't be tracked at all:

apiVersion: configs.flanksource.com/v1
kind: ScrapeConfig
metadata:
  name: terraform-exclude-secrets
spec:
  terraform:
    - name: '{{ .path | filepath.Base }}'
      state:
        s3:
          bucket: terraform-states
          connection: connection://aws
          objectPath: '**/*.tfstate'
      transform:
        exclude:
          # Exclude all AWS Secrets Manager resources
          - types:
              - 'AWS::SecretsManager::Secret'
              - 'AWS::SecretsManager::SecretVersion'
            jsonpath: '$'

          # Exclude IAM access keys
          - types:
              - 'AWS::IAM::AccessKey'
            jsonpath: '$'

          # Exclude specific sensitive fields from all resources
          - jsonpath: '$..master_password'
          - jsonpath: '$..auth_token'
          - jsonpath: '$..private_key_pem'
          - jsonpath: '$..client_secret'

Best Practices

• Leverage Default Masking: The scraper automatically masks common sensitive patterns - verify these meet your needs before adding custom rules
• State File Protection: Always encrypt state files at rest and in transit, use remote backends with proper access controls
• Use AWS Secrets Manager: For RDS instances, use manage_master_user_password = true to avoid storing passwords in state
• Regular Rotation: Rotate credentials regularly and update masking rules accordingly
• Audit Access: Monitor and audit who has access to state files and scraped configurations
• Test Masking Rules: Regularly verify that sensitive data is properly masked in scraped configurations