Evolving your Security Team and letting the robots do the work
If we look at the past 10 years technology has been shifting pretty dramatically. Our development, ops and data teams have all made pretty drastic shifts in how they work to improve velocity, reduce friction and increase output of their product. Your development teams had to evolve to stay competitive in your industry or your team / product is at a disadvantage.
We have seen the CI/CD pipeline become the enabler where everything can integrate to form a seamless experience from development all the way to production. Developers can write code and push it into the pipeline, where automated tests will run, the code will be containerized, and the resultant image will automatically be deployed on a Kubernetes cluster running an immutable operating system on a cloud provider that offers, for all intents and purposes, unlimited elasticity. The logs, traces and other metadata from these applications can now be stored in data warehouses and analyzed with cloud based data analytics tools.
One area we haven’t seen as much change is in the security area. Don’t get me wrong some teams have adapted but many have not, mostly because it is hard to see how to change when you have been doing things for the same way for so long or when you have compliance requirements you need to meet. These teams want their hard toll gates, change tickets, reviews, et cetera: very manual processes that break the end to end automation that other teams strive for. How do we change that and what does good look like?
First we need to change the security mentality into that of enablement. I have seen many security people be proud they are the blocker, the person that is known for “no”. I was especially shocked when I saw a security executive stand up at a Google Cloud event and claim he didn’t believe in cloud. That type of mentality can no longer be allowed in the security space and should actively be shunned, the question to ask yourself is “how do I enable my teams to go fast while providing the security/compliance I need?”. The answer, as one of my developer co-workers put it, is: “let the robots do the work.” To that end you should focusing on automating as much as you can and shift your security team to be a devops team focused on security automation and development.
So let’s start at where it all starts, your code repository and your CI/CD pipeline. Your development teams are already using them for automation so have you integrated your security tooling there? You can add your static code analysis, secrets checks, compliance checks, vulnerability scans, container scans, application scans, etc. all right in that pipeline and break the build in order to give your developers instant feedback right in the pipeline. You already know what you need to check from a security and compliance perspective and you know what hardening standards you need to meet, so turn those requirements into code and put them in your pipeline as automated checks.
Rapid automated remediation
If we are building everything as code (including our infrastructure) and ensuring things are secure in the pipeline then we know what good looks like. Anything outside of good is bad and should be terminated automatically. Why do we as security people want to continue the alert/respond method when many things are easily remediated in automated methods? Security teams are either stuck in their old ways, or as a colleague pointed out, they are so scared of breaking something that they want a human to push the button. However, humans make mistakes too, and there’s no guarantee that you (as a human) will always be able to respond to and remediate that alert before your data is exfiltrated.
The solution is rapid automated remediation and there are a number of tools like Turbot, Divvycloud or Cloud Custodian that can automatically remediate security issues in your cloud environment.
Below are some examples you can automate in your cloud environment but before you do that I would suggest coming up with a comprehensive tagging standard for your company and tag ALL the resources you can.
Example tags (key / value)
- owner / ross
- bu / security
- environment / production
- application / turbot
IAM (Note: You should be using SSO with temporary access keys https://github.com/Nike-Inc/gimme-aws-creds)
- Remove any user not attached to a group
- Remove users that are not whitelisted
- Remove any AdminAccess policy attached to a user/group
- Ensure password policies are enforced
- Remove stale access keys (30 days)
- Ensure all users have MFA enabled for console access
- Remove all untagged/mistagged resources
Storage
- Lock down public buckets that aren’t whitelisted
- Ensure bucket encryption is on
- Ensure bucket access logging is on
- Enable versioning
- Remove all untagged /mistagged resources
Instances / Networking
- Only allow instances of certain types
- Only allow instances in approved regions
- Ensure all instances use encrypted storage
- Ensure flow logs are turned on for all VPCs
- Remove any 0.0.0.0/0 rules that contain ports that you don’t want exposed like the standard ElasticSearch ports
- Remove all untagged / mistagged resources
(Note: There are many many other rules and cloud specific rules you can implement. Start with your top 10 and go from here.)
Implement these rules in dev, test, staging and production not only to protect all of your environments but so your team has consistency. This way, if there are any issues they will happen in dev before they hit production.
By implementing just these few automated rules you will solve most of the common security issues we see in cloud environments like open buckets, open ElasticSearch instances and issues around access keys. You will also prevent one of the most common abuses when a malicious actor gets access to a cloud environment and spins up a ton of expensive gpu instances for bitcoin mining (remember we only allow certain instance types).
(Credit to Don Huang for always helping me with my data and checking my articles)
