VCD RoundTable

Welcome to the VCD Roundtable Episode 47.

We want to talk about the importance of hardening.

So hardening software.

So we added the software at the end.

Oh, I didn't know that.

I thought...

The topic is around hardening, but OK.

If it goes around VMware products,

maybe I can help as well.

Hi, this is Matthias,

also a part of the VCD Roundtable stream again.

Today's guest speaker with us is Fabian.

Fabian, for the introduction.

Hello.

So I've been here many episodes ago.

I don't know, maybe one of the first and never came back.

But now if things get hard, I'm back in the team.

Fabian, architect at comdividion.

I do a lot around architecture and consulting service

providers when things get tough.

Nah, just kidding.

No, but the topic around hardening is something

I dropped in during our Kickoff

because we see that a lot of service providers

need to be prepared

against any kind of cyber threats

and attacks and also audits.

So be prepared for the audits.

That's why hardening is an important thing.

And maybe I can drop the one or other story

to give you guys out there some feedback around that.

Absolutely.

What's also interesting, based on the questions

we get from many customers and service providers,

is do we need hardening only if we need to get certified

or is it also relevant in normal environments?

Yeah, I mean, hardening is a general concept for many years

to change your software probably in a way

that differs from the default settings.

Yeah, there were many years ago

when... was telling you

how to harden your Windows systems

and all kinds of software can be

hardened by disabling features,

by also giving you guidance around operational steps.

You need to patch your environment.

You need to make sure people are trained.

All those things come together

when we talk about the topic hardening.

That's why I always see it as a three-part thing.

One is the architecture around solution.

That must be hardened.

One is the configuration, how is it applied?

Those hardening settings.

And also how is it lived?

So how are daily procedures hardened against security?

This is something you always need to observe

as a complete item from my opinion.

And if someone promised you,

"Hey, we deploy Aria Operations or VCF Operations"

and you have automatically your environment hardened

or get reports of the hardening status.

In my opinion, that's not true.

It's one important toolkit for it.

But there's much more to hardening

than just installing something

and saying, "Now it's hardened."

I wouldn't phrase it that simple

because honestly speaking using Operations Manager

and the implemented policy or the implemented framework

with the hardening configuration is pretty straightforward.

It just reflects Broadcom's default view:

which configuration should be taken

or which feature could be reconfigured

to have a hardened infrastructure

from a Broadcom perspective.

That is one of many approaches.

It might fit your needs.

It might not, but you get a proper

report based on Broadcom's default view,

how it should be done.

If you don't agree, if you have a different approach

on hardening your infrastructure

or you have different security

and recommendations/needs,

you just need to reconfigure the

framework within Operations Manager

to fit your needs.

Then you have a proper report based on your infrastructure

and you can start reconfiguring everything.

Yeah, totally right.

But from my point of view,

as you know Ops Manager has a limited view on things.

If I think about the environment of the service provider

who might also have NSX, Advanced Load Bancer

or I don't know Cloud Director in place,

that's typically something the

Operations Manager does not get.

If we talk about those, let's say hardening relevant items

like "Hey, your Active Directory must be secured."

Let's say, immutable against any kind of attacks.

That's also something we cannot.

So we have only this limited view

and I really love using those built-in compliance options

from Aria Operations as a starting point.

But still, and you did a great point by saying,

"Hey, it's everything related to risk

and everyone needs to decide fon their own

if they want to accept the risk of not applying this item

or maybe this item does not fit

for your specific use case environment."

And from my experience over the last years,

I did a lot of... I was involved in a lot of audits

around Enterprise solutions.

And the important thing is you

need to have something in place.

That's the first thing.

So you need to do this kind of hardening.

VMware has done a great job over the years

together with the community of

creating those hardening guides;

in the past they were called

security configuration guides.

Those big Excel sheets

where you have the default setting,

that's what it should be.

That's the argument why you should have that.

And what we always did, what we still do nowadays

is taking that extension for certain customers

and have the discussion -- item per item with the customer.

"Hey, how does it look like in your environment?

Is it a risk we take?

Shall we live with the operational downside

of enabling lockdown mode?"

Which obviously makes sense

from a security perspective in many scenarios,

but sometimes from operational perspective,

it can be quite annoying.

So we need to get the risks together,

calculate what's relevant for the customer

and then make a decision.

And I think that's a very important point you just covered,

because even though a guide tells a customer or a CSP

setting ABC needs to be reconfigured to whatever,

you can still come up with a decision

against the recommendations.

Say, "I'll configure a feature

in a certain way because of..."

And even if you have an audit...

so an auditor has no chance to make any...

He cannot force you to reconfigure it

to a certain configuration.

But if you have it configured differently

based on common guidelines,

you always need to have a justification in place.

You need to take a proper decision and justify

why is it configured that way?

And as long as you have a justification,

you're good to go.

Because if you have a justification,

you made the decision and you're perfectly aware

about the pros and cons.

Yeah, depends on the justification.

I don't encrypt because it's complicated.

[laughter]

Would be the...

I mean, from my point of view,

most auditors I work with, most of them,

they audit a broad area of IT.

They're not experts in VMware.

But they react differently

if they have a question and want to see something

or some documents, artifacts, whatever.

And you take something out of the box.

Here, that's a complete list.

It looks great with bullet

points and score and justification.

You show it to them.

You then take an extract from the Aria Operations report.

So they see there is a mechanism in place

to prove that certain configurations are in place.

This is something that really helps you

during the audit.

And for sure, for your own sake,

it's up to you to still make the best design decision

that secures you, but still keeps operations alive.

And this is really relevant from my point of view.

That's why, I mean, if you do those security engagements

or hardening engagements,

three to five days of architecture is easily gone.

You have one, two, three days of workshops,

then everyone goes back together and make maybe different

decision about those 10, 15 complex points.

You need to, let's say, also,

or try to get the proof for certain statements.

Yeah, maybe that the backup is encrypted.

Simply something you cannot see.

You need to assume that the backup is encrypted.

But still, when we go out there, I ask them,

hey, give me some proof or some configuration proof

that the backups in the back end are encrypted.

And this is really relevant.

So those engagements are really important,

but also take a little while.

And that's what I said.

You cannot simply deploy Aria Operations.

Say, now it's hardened and let it run.

You need to discuss that.

It's an architectural decision you need to take

around the environment.

And even if you have some configuration monitoring

in place, like Operations Manager, for example,

it's just a supporting tool,

because you need to make the architecture.

You take the decisions, you justify.

The monitoring is just a tool supporting you

to prove that you implemented what you have discussed

and designed.

That's one point.

But what we also often saw on the health checks,

for example, that there is hardening

or hardening was done, hardening was documented.

But then they had an issue and needed to open SSH

or direct access to hosts

again and never configured it back.

So from that perspective, I think, it is necessary

to have proper control of all mechanisms and changes made

during the hardening process to ensure they remain active.

And totally, totally true.

And one example that always comes to mind is,

for example, I'm a big fan of

segmenting the VMkernel ports

of the ESXi, onboard firewall.

We simply say only management networks can interact

with the ESXi VMkernel, which is easy to do

and gives you direct or reduces the attack vectors.

And even though there's some

malware in your infrastructure,

they cannot reach the ESXi network.

Except, I mean, you could, if

you have some within data center

firewall, you could do something similar.

But it's one way to make sure the packets don't arrive

from unknown networks at the ESXi.

And when I look at all... fortunately, over the last two

years, I've known many companies had

some issues with crypto attacks

on their VMware environment.

Fortunately, none of my direct customers were

attacked by that, or it was broken or encrypted.

But the primary way I got engaged afterwards,

it was always the same thing.

Some older version, non-patched version of ESXi,

they were reachable from one of the desktops

that had some malware installed.

So client network, server network,

server network, they could scan the network,

figure out ESXi host, figure out older versions,

and did some attacks.

And the other perspective was always

around this active directory integration of the ESXi host.

So also one thing, when the ESXi host was integrated

in Active Directory for use authentication,

there were a lot of attacks happening over this path.

And today, I think in the hardening guide for security,

there's still the recommendation

to add the ESXi host to the Active Directory,

to have a proper directory or non-route-based log on

as possible, even though that's the official statement.

And you also see in the CIS guidelines

for security or other documentation

when you want to reach a certification,

I always argume," hey, let's not do that right now."

There were so many bad things happening over this way.

We do log them out.

We segment the ESXi from the network,

and we make sure we have some alerting in place

when the root user is used on the ESXi host.

So you have Log insight configured, making alerts

if roots are doing any actions

out of our role-based authentication mechanism,

because that is really important

to make sure every action is accountable

to a specific user or system.

But that's... all the topics you've mentioned.

They're all part of a proper hardening,

because hardening is not a tool.

It's not a solution.

It's a process, right?

It's something--

It's a lifestyle.

It's a lifestyle, right?

That we're not going down there.

Yves is not here, Fabian is here, and we're drifting.

So it's a process you take.

It's an approach, and also monitoring,

and many different things that come to your mind

might be part of a hardening process.

Like you said, if a user is used to log into a system,

or you mentioned the AD thing with the ESXi host,

it's also pretty fine.

But I think even if a recommendation exists

and you come up with, "no, I

don't think that this is a good idea,"

like joining an ESXi host to the Active Directory,

you just justify it.

The downside of it is, and just being really blunt,

is there are many auditors out there without any clue--

No, I should not say without any clue.

That's not good.

Not being in depth with a certain product behavior.

We'll tell you in the report, your ESXi host

is not joined with Active Directory, so you get a minus

score for something which does not make any sense

because they have no clue.

And then if they have a good name,

if the company has a very well-known name,

they are always right, even though they are wrong.

Absolutely.

But the good thing is, those are the items,

even if everything's correct,

those are the things they can find.

But they feel good.

We feel good.

But still, the Active Directory thing,

and if anyone is here

listening, I would be really interested

how you are dealing that around the ESXi hosts,

because I have a clear opinion just based on what I saw.

Because I saw things, and I don't want to see them again.

I think if you're coming up with your approach

and how you would configure infrastructure,

I think as long as you have,

as long as you take the decision

document the decision, that's the proper justification,

you're good to go because that documents

that you're perfectly aware of what you're doing.

I think that's one very important point

around the whole hardening and those topics.

True.

Sasha, how can Cloud Foundation help us with hardening?

I want to hear some sales talk, come on.

Yeah.

It's the best solution.

Absolutely.

So you get a lot of configurations pre-configured by VCF,

so that's interesting.

But I think the one topic also for me,

a big part in hardening is to work with certificates.

Work with public certificates,

make sure that you have a trusted CA

behind your certificates and not working with

self-signed certificates out of the host.

And having a hardened certificate authority.

Now we start again.

Yeah, I mean, that's also a thing.

Absolutely.

It doesn't matter if your

certificate authority is compromised.

That's a huge thing.

I always get the feeling when you go down the security

in Hardening Road, it's opening the box of Pandora

because from one item, you get to the next one, ask,

hey, what kind of CA certificate authority do you have?

Is that something we can

assume as being reliable and secure?

We don't know to be honest. Especially if you talk

to smaller corporations where

that was set up by, I don't know,

Erwin and Dieter.

And they left the company 20 years ago.

You know, Erwin and Dieter, that's a common German name,

just for Mike and Jane, if you're from the US.

But still, this is something where you then also say,

okay, perfect.

When we are done with our, I would say, assessment

or hardening sessions, and everyone is really hard.

No, that's not the right thing.

And everyone is really getting items to work

in the VMware infrastructure.

There's also a lot of to-dos to figure out

if we assume the right things around Active Directory,

certificate authorities, and other topics.

Erwin and Dieter.

Come on, you said we can cut this later on.

So therefore--

Well, we'll let it slide.

So, cool.

But again, it's also the combination

with the monitoring tool.

Because you need to have a report,

because we are all just human beings.

And if we are tasked to configure something,

and even though it's very well prepared and documented,

and we receive a monkey, see monkey, do instruction set,

there is still the chance that

you're fat-fingered something.

And that's the reason you need to have a pool

in the backend, monitoring the infrastructure,

being aware what should be configured,

and compare it with what is configured.

And, Sascha, I love the example you came up with earlier on

with the SSH.

So that's a common use case, like,

"oh, I need to troubleshoot something."

I enable SSH.

What is perfectly fine, right?

And at the end, because you disabled all the warnings,

because they drive you bonkers in the UI,

you forgot to turn off the SSH service

at the end of your troubleshooting session.

So latest and next day,

the compliance report within the monitoring tool,

which is Operations,

you shall have only one Operations Management tool,

pops up saying, "I found this...

"design and deployed by comdivision"

Yeah.

... here is a new host, and this host has

SSH turned on," and then you receive an alert,

saying, all right, I detected a configuration drift.

Please help me turn off SSH.

And even though you have an incident,

so you violate one of your policies,

but if you have a monitoring tool, you get an alert,

and then you can react on, like, "oh, I forgot.

I turned the SSH servers off."

That's a proper approach, and that's perfectly fine.

The worst thing is you are not

aware of that a configuration

drift happened, and you are not

compliant with your old policies anymore.

That's the worst thing.

Yeah, totally something we need to keep in mind.

I mean, the good thing is over the years,

the core VMware products, and

if you now look into the security

configuration guide for vSphere 8, there's not so much more

in it anymore.

The product team did a great job over the years

to bring the core hypervisor to a

level where you should typically,

or you don't need to tweak that much.

I think Mike Foley, who was also in the product team,

he's very often shouting out on LinkedIn that he

was very keen after getting the hardening into the product.

We see that VMware is getting

better and better in a lot of areas.

That will also be much more better than with VCF 9.

I'm pretty sure about that because

now they all try to make it secure

right from the beginning.

But again, there are a lot of things we need to decide.

Second factor of indication, for example,

that's nothing you can bake into the product.

We need to make a proper decision, and

then also how to configure it concretely.

Even if that's not a real word, I think.

But also Cloud Director.

I mean, we are big old fans of good ol' Cloud Director,

and also there are still a lot of

things we need to tweak afterwards

to make sure it is hardened according to the recommendation

that VMware put it out also many years ago.

Maybe not Broadcom.

Maybe the recommendation which

comes from either comdivision

or even better from the customer themself.

Because that's one thing we do with our CSPs and customers

is interview them.

What's your expectation?

What are your business

requirements towards security hardening?

And those bits and pieces.

Another interesting topic.

Let's move away a bit from the base infrastructure.

And let's maybe for the last few minutes

focus a bit more on the workloads which are running

on the already successfully hardened base infrastructure.

So we have guest operating systems.

We have applications and a ton of stuff running.

And if we are now back into the CSP game, it's a CSP.

You need to make sure that you

have proper network implementation.

Because you have no clue what your

customers are doing inside the virtual machine.

They install the guest user and you just don't know

if the customer applying all the

security patches as recommended

by the operating system vendor for example.

Yeah.

And remember those site channel attacks a few years ago.

You know where they said, "Oh, disable hyper threading."

Because there were chances of

whatever this is something that could be.

Could be a real...

So winning the lottery had a higher chance than that one.

But...

I'm not sure.

I mean there were attack patterns

where you could extract certain things.

And this is something you

cannot control as a service provider.

The thing here.

What was the name of it?

Spectre and Meltdown.

Winning the lottery has a higher chance than that.

No, I prove it wrong in the show notes.

Do we have show notes?

We will get some.

So I can tell you on the side why.

But the thing is you have your...

And that's not a guest OS thing.

It was a hypervisor attack.

So two different things.

But you have no clue what your

customers are doing inside the virtual machine.

And if they're patching their OSs,

if they apply security patches to the applications.

And we are perfectly aware, because

virtualization is a big invitation to keep

expired software running year after year

after year out of support, out of everything,

not supported anymore.

So many companies do that.

So as a service provider, you never know

what's happening inside the infrastructure.

So it's even more important

from our perspective to take care

that the underlying infrastructure is properly secured.

And also you are logging all the

traffic between tenant networks

and the CSP basic network infrastructure.

Separate.

And additional to that and additional to hardening.

So really take a look what

options you have to help your customers,

your end customers, to bring more

security inside their environment.

So take a look at advanced threat protection with NSX,

which you can buy as an add-on.

So that's one point.

And the other point is take a look at advanced ALB.

So advanced load balancer.

So for all the web services your customers hosting.

So you can activate web application firewall on top of it

and make your environment or the

customers' environments more secure.

So ATP, so I first mentioned

ATP, that's a different ballgame.

It's an amazing feature.

It's there and even with DPUs it gets usable.

Because you're offloading everything to the smart name.

And use those features because I haven't.

So I've been in IT for more than 25 years.

So over time, year after year, you just see more different

different patterns.

It got more and more and more.

There was no year where you had less

threats running around the internet

than the year before.

So it gets worse and worse.

Okay.

And then in the end, you always need to have

Operations in the back end for monitoring.

Awesome.

So if you are not hardened already, what can we do?

Sascha, how much is our hardening?

Standard offering.

What do you say?

You need now to turn on the red light.

And then...

Wait a sec.

So if you need...

Not red enough, I would say.

So if you need it really hard.

No, wait a sec.

Guys, you are aware that it's a podcast.

Most people won't see the video.

Really?

We're now waiting for the audio description.

That's why we talked about the red light.

Yeah.

So for all the audience who's not watching the video,

the light in my room is now pretty red.

And joking aside, if you need any guidance around that,

if you feel your VMware

environment could use a third party look up

around the topic of security

hardening, just drop us a message

and we can make you pretty quick and offer to help you

nearly on all continents nowadays, isn't it?

Sure.

Yeah.

Feel free to ping us.

Feel free to drop us messages on

LinkedIn, as an email, over our webpage, etc.

and we can sit together and talk

with you about what we can do for you.

How can we solve your problems and provide hardening

or a design for hardening for your environment?

Yeah, also from my side, thanks for tuning in.

Famous last words, I think hardening is very important.

Also monitoring and implementing

the stuff, but always keep in mind

that you keep your risks and hardening

aligned with your business requirements

to have a solution because in the

end you need an infrastructure solution

which supports the goals the business has.

Otherwise it doesn't make the best sense out of it.

Short outlook Ep. 48.

I heard a rumor.

I heard a rumor that Episode 48 will

be around Orchestration Automation.

Yeah, I want to come back.

So that's the rumor.

So we won't discuss VCF

Automation because that's not released.

So Orchestration Automation will be

around Cloud Director and Orchestrator.

That's the plan.

Interesting.

I'm looking forward to this one.

Who will present, not me.

I'm sure you.

He's Mr. Orchestrator.

Come on.

I know.

Okay, thanks for tuning in.

Have a great day.

We're more than happy to help just ping us.

See you in the next episode.

See you.