Mastering AWS Resource Visibility & Governance with AWS Config

In the fast paced world of cloud computing, especially in Amazon Web Services (AWS), resource management is key to operational excellence and financial prudence. As more and more workloads are migrated and scaled on AWS, the ability to monitor, control and optimize cloud costs is a key differentiator.

This Article will cover the foundation of AWS cost optimization with a focus on using AWS Config for resource monitoring. For FinOps, FinTech, DevOps and AWS enthusiasts, mastering these techniques is the key to getting the most out of the cloud.

Table of Contents

Understanding AWS Resource Management

Effective AWS resource management is the foundation of a cost efficient cloud strategy. It’s a continuous cycle of monitoring, analysing and adjusting resource allocation to meet performance demands without unnecessary spend.

AWS provides a set of tools to help with this. AWS Cost Explorer stands out with its detailed reporting and visualisation features for AWS spending. It allows you to drill down into costs and usage by various dimensions, including tags, services or time periods.

This level of granularity is critical to understanding spending patterns. For example, you can create custom reports in AWS Cost Explorer to monitor specific resources, such as a fleet of EC2 instances or Amazon EBS backed storage (like EBS volumes) and adjust resource allocation for better cost efficiency.

Beyond native offerings, the environment supports niche tools. Consider Chaos Genius, for instance, an open-source tool that integrates with environments like Databricks (often hosted on AWS) and leverages machine learning to detect anomalies and unusual spikes in cloud spending. Such tools complement AWS offerings by providing proactive visibility.
The very character of AWS in being capable of accessing elastic and scalable cloud resources is at the core of managing and optimizing various workloads. Cloud environments like AWS facilitate hosting virtual space and data processing necessary for resource management and general operational efficiency.

The problem is how to leverage this elasticity without bearing prohibitive expenses, particularly with compute resources.

Importance of Cost Optimization in the Cloud

Cloud cost optimization isn’t just cost cutting, but finding the right mix of cost, performance, agility, and resiliency. For the serious cloud users, it’s an ongoing cloud financial management practice. It’s essential to ensure long-term success and sustainability.

One of the key tactics in establishing cost-consciousness is implementing chargeback or showback models.

Chargeback frameworks :enable companies to charge back cloud expenses to departments, business units, or projects. Such direct accountability promotes prudent consumption and optimization efforts by these teams.
Showback models: while not billing directly, are transparent because they report the costs to the respective departments. This transparency allows the departments to recognize their cloud usage trends and areas where they should optimize.

Proper cost allocation is dependent on comprehensive resource tagging. Resources can be tagged with metadata like Project, Environment, Owner, or Cost Center to create granular cost reports that are necessary for efficient budget management.

Azure Cost Management (for Azure environments) and AWS Cost Explorer are tools that can break down costs along resource tags to enable precise expense tracking to different organizational segments.

Finally, chargeback and showback programs, driven by strong tagging, promote cloud resource utilization, resulting in possible elimination of waste costs and a more financially sustainable cloud stance.

Flow Chart Of departmental reports on aws config

AWS Config Overview for Resource Monitoring

AWS Config is a core service that allows you to audit, monitor, and review your AWS resources’ configuration. It gives you a complete inventory of your AWS resources, their configuration, and how their configuration evolves over time. AWS Config is something that one should learn and implement in order to reduce cost by making sure that resources are configured in an optimal way.

Key capabilities of AWS Config include:

• Continuous Monitoring: It enables tracking of AWS resource configurations and their relationships over time, providing a comprehensive historical view.
• Configuration Data: AWS Config gives a detailed picture of the configuration of AWS resources in your account and their relationships with each other.
• Compliance Checking: You are able to utilize AWS Config for checking that the configurations of AWS resources are aligned with best practices and internal policies using Config Rules (AWS-managed and custom).
• Operational Auditing & Security Analysis: The service should facilitate operational auditing, security analysis, change management, and troubleshooting.

By constantly monitoring and assessing resource configurations, AWS Config is a key enabler of operational stability, security, and governance throughout your AWS infrastructure.

Discovering Unmonitored AWS Resources using AWS Config

“Unchecked” AWS resources – resources deployed without governance, monitoring, or lifecycle management – are a typical cause of idle cloud spend. AWS Config helps to detect these resources by offering an organized way of tracking and monitoring them.

With AWS Config, users can define rules (Config Rules) to automatically assess the configuration of AWS resources against desired configurations. The rules may either be user-authored or leverage a broad set of AWS-managed rules.

With these rules defined, AWS Config is able to identify resources that can be idle, are non-compliant with security, or are not cost-optimized configurations.

For instance, a Config Rule can verify whether EC2 instances are tagged as being an “Owner” or whether S3 buckets are configured for public access. A rule can also highlight EBS volumes that have been in the “available” (unattached) state for an extended period. Merging AWS Config findings with cloud cost optimization workflows or tools can detect duplications and streamline resource spending.

AWS Config’s feature of delivering configuration history complements the role of identifying unused resources as potential cost-saving opportunities by offering a measurement of the duration for which a resource has been in a potentially wasteful state.

Setting up AWS Config

Configuring AWS Config is a simple process, normally consisting of:

1. Activating the Service: Activate AWS Config in the desired AWS regions. It is generally recommended to activate it for all regions in use.

2. Resource Type Selection: Define what types of AWS resources you wish AWS Config to capture. You may choose all the resources that are supported or individual resources.

3. Set up an S3 Bucket: AWS Config saves configuration history and configuration snapshot files in an S3 bucket.

4. Creating an SNS Topic (Optional): To get notified about configuration change or compliance status change.

5. Config Rules Definition: Here’s where AWS Config’s detection and governance capability comes into play. Users can choose from managed rules that are already defined or define their own rules using AWS Lambda.

6. Auditing Dashboards and Reports: AWS Config offers dashboards to display resource compliance and configuration information for auditing and compliance checking in AWS environments.

After you’ve got it configured, AWS Config starts auditing the configurations and scanning them for compliance with your specified rules, delivering notifications for resource non-compliance or configuration changes.

Configuring Resource Rules

Setting up resource rules in AWS Config is the cornerstone of automating the discovery of unchecked or non-compliant resources. Unchecked AWS resources can result in undue expenditure if not discovered and remediatively addressed accordingly.

Examples of how rules help:

Compliance: AWS Config can be used for monitoring the configuration of AWS resources and assisting in complying with internal or external compliance (e.g., making sure encryption is enabled for S3 buckets).

Idle Resource Detectors: Not necessarily “idle” detectors, but rules can detect setups that are commonly found alongside waste. For example, a rule would detect EC2 instances running older, less efficient instance types.
Lifecycle Management Triggers: AWS S3 lifecycle policies can be set up to automatically move data to cheaper storage classes (e.g., S3 Intelligent-Tiering, S3 Glacier Instant Retrieval) after some time. A Config Rule can verify whether lifecycle policies are set up on buckets. This also addresses data transfer cost management in case objects are moved across regions or out to the internet.

Underutilization Hints Identification: Idle resource detection can be utilized to identify underutilized resources for decommissioning purposes to rightsize cloud expenses. A Config Rule can be used to scan for load balancers that have no healthy backend targets.
Automated Cleanup (Indirectly): Although Config mostly finds, its discoveries can initiate automated remediation activities through AWS Systems Manager or Lambda. Automatic cleanup of unused resources generally entails establishing rules (in Config or somewhere else) that specify when resources are to be flagged for investigation, transitioned into a state for deletion, or deleted outright.

Conclusion

Establishing robust resource visibility and governance through AWS Config, as detailed in this blog is the foundational step towards mastering AWS cost optimization. By understanding your resource landscape, tracking configurations, and ensuring compliance, you build the necessary groundwork for more advanced optimization techniques.

FAQs

What is the AWS Config used for?

AWS Config is a service that enables you to assess, audit, and evaluate the configurations of your AWS resources. It helps track resource changes over time, maintain compliance with desired configurations, and identify security or operational risks by providing a detailed inventory of your AWS environment.

2. What is the difference between AWS Config and CloudTrail?

While both AWS Config and CloudTrail track activity within your AWS account, they serve different purposes:

AWS Config records the configuration state of resources and how they change over time.
AWS CloudTrail logs API activity and user actions within your account for security auditing, troubleshooting, and compliance.

In short:
Config = “What changed?”
CloudTrail = “Who did what and when?”

3. What is the difference between AWS CloudWatch and AWS Config?

These two AWS services serve different but complementary monitoring functions:

AWS CloudWatch monitors resource performance and operational health, using metrics, logs, and alarms.
AWS Config tracks resource configuration changes and compliance against set rules.

In short:
CloudWatch = Performance and operational monitoring
Config = Configuration state and compliance tracking

4. What is the use of Config?

AWS Config is mainly used to: