Cybersecurity Analytics Roundtable Discussion, Trends and Advice

The following is a transcript from a panel discussion on critical changes in cybersecurity analytics, trends and advice. The session was presented by, elite ML consultancy, as part of an ongoing series about machine learning (ML), with panelists including Cazena, offering fully-managed Big Data as a Service. The complete discussion is available as a video via The transcript has been lightly edited for readability. 


ScottShagory.jpeg ScottCohen_3_225x225.png LovanWeb.jpg Steven Shack (1).png

Scott Shagory
Strategist and Technologist,
US National Weather Service

Scott Cohen
Managing Partner,

Lovan Chetty
Product Management, Cazena

Steven Shack
Data Scientist,

Transcript: Cybersecurity Analytics Roundtable Discussion from April 11th, 2018

Scott Cohen: Hello everyone. Welcome to Big Roundtable. My name is Scott Cohen, managing partner of I'm joined here by Lovan Chetty, head of Product Management for Cazena.
Scott Cohen: And Steven Shack, who is a principle data scientist with Shortly, we’ll be joined by Scott Shagory, Strategist and Technologist to the US National Weather Service.
Scott Cohen:

Today's topic is the application of machine learning for cybersecurity. I think it's a pretty hot topic for a lot of large companies. For this roundtable, we intentionally did not plan very much. We're hoping to spark a free flowing conversation amongst the panelists, and really urge all attendees to raise your hand and ask questions. We can promote you to a panelist, and you can join in the conversation, if you are interested.

At a high level, just to kick it off, what we're talking about is: What are the applications of machine learning for cybersecurity that actually work? There's lots of ideas about where we could apply it, but what are we seeing today that is actually working? And from my understanding, it's very simple things, like detecting malware and spam. 

Really, the crux of the problem, which is true across all of machine learning, is that you need to have examples in order to learn from and train your models, so that you can detect things like anomalies and help support a human effort to cybersecurity. In general across the board, what machine learning is doing today is supporting humans. So, trying to reduce the false positives and get the human's time optimized. Because they're still going to be needed, at least for the foreseeable future.

Lovan Chetty  We've seen something similar with anomaly detection. We've got a few customers that are looking at it more from the social engineering perspective. Most of the big breaches that have been publicized have usually been because somebody was convinced to hand over their credentials. One of the problems they're trying to solve is I can't just look for that password coming in, because the person is going to have a valid set of IDs. But how can we tell that it's not one of our employees or authorized people that can do that?
Scott Cohen: I heard a radio segment on this subject the other day, and the subject matter expert was suggesting to think of everyone as compromised. So, consider everyone in at work, every device, every person as a threat. And then whittle away at clearing them as being ‘not a threat’ based on the patterns and activities. 
Steven Shack:  Google is pushing full steam ahead with their “zero trust” initiative. They're redeveloping their network, assuming that everything they have is directly connected to the internet, completely open, and then going from there. It is exactly what you said, but taken to the absolute extreme.
Scott Cohen: So, Steven, to put you on the spot, how can you determine what is normal and what isn't?
Steven Shack:  Well, you can do it by subtraction, through something like anomaly detection. Or you can use something like a two factor authentication that you can't fake, like a key fob or authentication device that has be kept with an individual. So, those are a few ways to do it, through attributes, anomaly detection and subtraction, and through positive identification.
Scott Cohen:  Let's get a little bit more into the mechanics of how this works. If you can see the screen share here, we have a workflow that we put up here that really applies to almost every machine learning application that we're building, which really run the gamut from e-commerce applications to other areas of financial services and insurance. 
Scott Cohen:
The key here is to understand the flow. This [above] is an acronym. It spells CRAVE. 
Collect: The first step is to collect data.
Reformat: The second step is to reformat the data so that it's useful for a machine to read and understand and learn from.
Analyze: Then you have your Swiss army knife of approaches that you can take and models and techniques for the analysis step, and that's really the hardest part of all of this.
Visualize: Then you see what you have and try to reduce the dimensionalities so that a human can view it and be helpful in assisting in the process.
Execute: Then execute. 
Scott Cohen: Steven, as we've dug into applications in cybersecurity, we see some approaches that are working, some that aren't. As far as I understand, there's no unsupervised learning that's working yet. Where do we see some effect of, perhaps, supervised learning applications?
Steven Shack: Network monitoring -- in terms of anomaly detection in networks. Also using it [ML] with graph theory to see "Where is this guy coming from?" And malware detection, spam has been historically a great place for supervised learning, because it's easy to get examples. Those are the three main areas that you're seeing a lot of progress directly right now with supervised learning methods.
Lovan Chetty: 
We had an interesting example along those lines: One of our customers tried to do something like that and they started getting a lot of false positives, just because of the nature of their business.
A lot of their staff travel all the time, so the location thing started to really cause them issues. So, it's interesting -- what they brought in to help that analysis was their expense system. Because, if somebody's going to travel somewhere, they have to purchase a flight beforehand and things like that. By bringing in that expense data, they'll be able to now have a few extra dimensions that they can start to look at in trying to eliminate some of those false positives.
Scott Cohen:
That makes a lot of sense. That's interesting. Before the show started here, we were chatting about the similarities of cybersecurity approaches with machine learning as they relate to e-commerce and understanding a customer journey. So, we're talking about things like next best action to get a customer to move down the process towards buying something, and deciding what's the right ad to send them, what's the right communication in general, what's the influence that they need to get them over the hump to the next step.
It is the same kind of thing here -- you want to see the patterns. You want to understand the steps and the journey and the habits that people have. I know an initiative out of MIT [Massachusetts Institute of Technology] that understands your interaction with the keyboard and how quickly you're interfacing with the keyboard, your mouse patterns, how quickly you can enter your password. How often do you forget your password. These habits become really important.

[Scott Shagory joins the panel]

Scott Cohen: Hello, Scott.
Scott Shagory: Hey guys. Apologies for being late. Last minute crisis that finally got averted, so I was able to join.
Steven Shack: No problem, well you've missed out on a lot of wisdom here.
Scott Shagory: Hey, you know, there's always a price to be paid for being late, and that's fair enough. Guilty as charged.
Scott Cohen: Okay, Scott, well, since you're late, you have to introduce yourself.
Scott Shagory:
Good afternoon everyone. Or good morning if you're in an AM timezone. My name is Scott as well. I work for Raytheon, I do some strategy work for them and for my client, which is the National Weather Service.
That's how I met Scott here, through my blog, which I focus on core business strategy in the internet of things. The blog has morphed itself a bit, given the whole conversation around IOT and security to include also interviewing start ups in the cyber security space, as well. And then talking with folks related to my interest in blockchain-related technology and platforms. There's a big conversation, obviously, around that, given that the whole foundation of it is based on crypto.
That's really, for me, the foundational area where I get involved. I interviewed Scott for BigR last year, and he was kind enough to invite me onto the panel. So, hopefully I'll be able to, while late, impart a few words of wisdom, if not I'll do the best I can.
Scott Cohen:
All right. Thank you, Scott. Let's get back to the screen here [CRAVE graphic]. So, looking at the steps involved in machine learning. First step is to collect.
Scott, your intro triggered a lot of thoughts. You're talking about things like weather and how that could impact threat detection or how that could impact cybersecurity in general. It's somewhat orthogonal data set, but it brings up the point of “more data, more better.” That's how I like to think of it. That's a shout out to Honey Boo Boo [TV show character]. She said “more butter, more better.”
Anyway, we just talked to a large financial institution. They're doing cyber security, and they're using machine learning for it. What they were really interested from us is using our natural language processing capability to label data in real time. [That means] natural language as it relates to things like weather, perhaps. Maybe that's a stretch, but maybe more pertinent to this conversation are things like social media data, and other forms of information outside of the typical “cybersecurity” realm…data that might impact decisions as you progress.
Lovan Chetty: What would be an example of something they would look for in social media?
Scott Cohen: We were just talking about habits and location. Social media might gives clues to that. Especially current tweets might give an indication of what someone's thinking about or doing. If they're real or not, maybe?
Steven Shack: A little bit of a controversial indicator, but ethnicity detection. So, if you're a white or a black American male, you give off a lot of verbal and written cues in your language, whereas if you're say, a Russian hacker trying to generate messages and such on social media, your language is going to be slanted slightly differently.
Scott Cohen: 
Natural language processing in general seems an important facet here to understanding the bigger picture…because you have your transactional data, which is important; You have lots of history there. And, earlier in this conversation we were saying that in order for these machine learning algorithms to train, they need labeled data. They need the experience of history to learn and predict the future.
A lot of that is missing for this bigger spaghetti mess of data coming in. I don't think anyone's really stitched together social media to cyber security yet, to any great degree. And I'm probably mis-speaking here, because I'm sure the intelligence community has. I'm talking as the private sector in general. 
Scott, does that trigger any thoughts around we can marry lots of different data sets together for this collect step?
Scott Shagory:
Yeah, it raises lots of good questions. The things I know about in general commercial space, with respect to say Twitter or social media is instead of using a date as a trigger point, maybe you’d look for mention of a popular hashtag, or an interest rate [to take action]. Within intelligence community, there are probably lots of things that they do that we're not always privy to. I don't do any black box work myself.
But, yeah, I think it highlights a challenge.  What have really been the success stories for cybersecurity have been supervised data – it is malware, it's spam. I think of my personal experience where five years ago I would have to check my spam filter or spam folder quite often, whereas now I don't even do it maybe once a month, and there's hardly ever anything there.
Whereas a lot of this data is really all over the place in terms of its potential, as you pointed out. But to try and manage that, or to even deal with it, is just a huge challenge. 
We, or technically, our longtime client the National Weather Service have experimented a bit with social media data. We certainly share a variety of forecast data and other things with the public -- so that they're aware of it as well. It’s one additional mechanism, or way of sharing information.
But it is interesting to try to figure out how would we marry some of our prediction models with other data and use that in a way that would be helpful to the public. Our main mission is to protect the lives of the American people, as it relates to weather and their property. We've had a variety of conversations around it [social media], but it's been a real challenge to come up with something that is meaningful, that's not biased, that actually looks beyond gibberish, and people can really do something with. You want an outcome and you want people to take an action -- as opposed to just going "Wow, that's statistically interesting, but it doesn't really help us make a decision." And that's what the big picture should always be about.
Scott Cohen:
Right. There's a lot of statistically interesting data points in the cybersecurity world, but I think the key is to get that .01% that are actually, potentially, really a threat. That's the biggest problem here.
You started to talk about failures, I think, or you alluded to the idea. People are trying to apply machine learning, because it's the latest craze. But typically the failures are outside of the applications that you mentioned, Scott. 
Malware and spam is somewhat of a solved problem, but threat detection is a moving target, and very complex, because there's a lot of bad people out there that are trying the same approaches on the flip side.
It's interesting. One of the articles I read in preparation for this round table talked about zen and the notion that as a monk, you spend a lot of time looking at riddles and thinking about how that riddle affects you and your outlook on life. They then tied that into cybersecurity, in that there's so many edge cases and moving parts - it takes a while to figure it all out. And just when you do, it's changed again.
Lovan Chetty: Yeah, I think one of the trends we're definitely seeing with a lot of our customers is just their organizations get more complex. It used to be everyone that interacted with the [organization’s] data was an employee. But now they have a myriad of contractors and third-parties that provide services.

So, you've got this exponentially more complicated organization now, that they have to somehow give the right access to, which adds a whole other level of complexity to this problem. It was hard enough when everyone was in the same building, but now you've got people scattered all over the place, and they're not always an employee.
Scott Cohen: And threats are often within, especially for large organizations. So, where do we think there is promise? Where do you think the next step is? So, say spam and malware is pretty clearly an area that machine learning is improving our capabilities around. Where's the next ...not frontier, because I really want to stay practical here. What's the next step to having some real proof that there's merit to applying machine learning? Scott or Steven?
Steven Shack: I think in risk assessment. Not necessarily in online, active detection, but in risk assessment, we can definitely use machine learning to balance the outcomes versus the probability of events, in order to go through a network and find the vulnerabilities in a network, using machine learning based on past models. We can build up databases of this sort of thing. I think we'll see some good progress  in risk management in the very near future, if it isn't already being used in Google or other large organizations.
Scott Cohen:  Well, I know it's being used in financial services.
Steven Shack: That particular approach?
Scott Cohen: Well, the application of deep learning on risk analysis.
Steven Shack:  Oh, yes. For the actual finance. But in security, I think we'll use machine learning, these sorts of approaches, for actually modeling where the risks are. If we actually model them, then you can allocate your resources appropriately, rather than just doing a blanket approach of trying to secure everything.
Scott Cohen: I see. Okay.
Scott Shagory: 
What I would add, I mean, the two biggies that I tend to think about would be anomaly detection, of course, and the other one, to kind of pivot just slightly on Steven's comments, would really be risk-scoring.
So much of the cybersecurity industry [the big players, which heavily impact lots of other people's behavior in the space] has really been focused around peripheral defense or some sort of centralized network control.
It's always been very reactive. We study past attacks, or we study the past in general, but it's not a more proactive, predictive stance, however we want to approach that. And I do think the whole mindset is therefore very, again, defensive and reactive, which is why the industry as a whole is very ripe for disruption. 
That's why I like talking with startups about it. But I think more specifically, risk scoring is an interim step to encourage organizations to think a little bit differently. Whether it is natural language processing, or there's some other deep learning mechanism, use it as a way to start reshaping the entire conversation around what security should be. That would definitely be a step in the right direction.
Take the big data space, very broadly speaking. There's a big cultural component to this, but the conversation is starting to transition. Nevertheless, I often think there's a big disconnect, and I think this scoring piece might be helpful in that area.
Scott Cohen:
Sure. That's part around NLP. The general language that people typically use is what Google, Amazon, Facebook, et cetera, are all focused on. And general language, unfortunately, doesn't really help cybersecurity. It's really the main specific or the contextual data that you can glean from this myriad of data sources. I'm kind of transitioning us to the second step here, reformat. [CRAVE slide.]
This is where we need to make some sense of the data that we're getting and pipeline it in to a model, to try to make some predictions from.
But that reformatting is really important in as far as labeling it with semantic meaning, but also understanding the context of that data as it relates to threat detection or anomaly detection, or whatever it is that you're trying to accomplish. Does that spark any thoughts from anyone?
Steven Shack: No, I think that's perfectly accurate. 
Scott Cohen: And that's where a lot of people are trying to further the art.
Steven Shack:  I will say these collection and reformatting steps, they're where the bulk of the effort now is going. 
Scott Cohen: The analyze step was the hardest part, but the bulk of the effort, as you're saying, is the step prior. The majority of the work goes into the steps prior, but the real secret sauce, if you will, comes in the analyze step.
Steven Shack:  Yeah, the collection and reformatting is grunt work. It's messy, dirty, nasty stuff and no one really loves doing it, but it's necessary. But where the magic comes in is the analysis, and as you're saying, it's the more difficult part. 
Scott Cohen:  I happen to know of some initiatives to use deep learning for just those steps. Forget about the downstream predictions, just let's use AI or machine learning to do the collect and reformat steps.
Steven Shack: Interesting. My immediate thought is that sounds like an attack vector all of its own.
Scott Cohen:  That may be the last topic of conversation here, is where do we go too far and how does it backfire?
Lovan Chetty: I think that's still a lot of complexity in the “collect” step. Especially in large organizations where all of these data sets are totally siloed. They're siloed organizationally, so the complexity in getting the data is not a technical problem anymore. It's typically either political or an organizational problem.
Scott Cohen:
Sure. Well, if only there was a data sandbox in the cloud that could completely provide a platform, turnkey, as a service. I'm giving a shameless plug to my partners here at
Okay, any questions from the audience? I think we're at the Q&A point, unless anyone has any last thoughts they want to add before we transition?
Steven Shack:  Actually, maybe I will ask about execution. I mean, how do you link this back into an organization at an organizational level, rather than just a monitoring and reporting dashboard?
Scott Cohen: Well, I think that's just it -- whether it's cyber security or next best action for e-commerce and sales in general…The data science practice area within large organizations is still somewhat experimental and they have to prove that there is merit.
Steven Shack: I think what I would add is that it's a very difficult social problem there to get ... You can have all the dashboards that you want, but unless people all the way down the organization are actually understanding and appreciating and using what's coming out of that, it's not worth anything.
Scott Cohen: Right. Yes. 
Lovan Chetty: In some of the examples we've seen, they just skip over the dashboarding phase. So, the example there was if they could detect an anomaly, they would turn the individuals LDAP account inactive.
Scott Cohen:  That's a harmless step.
Lovan Chetty: It's a harmless step, because you're not destroying anything, but you're making it a little bit harder for that person to log on. So, even if they've gotten a bunch of false positives, you're going to have somebody that's a little bit irate, but…
Scott Cohen:  That's a way to force compliance!
Lovan Chetty: Yeah.
Scott Cohen: 
I like that, actually. Okay. Oh, wait, we do have a question. How is cybersecurity affecting the bottom line? Is it just about risk reduction, or could it become a competitive differentiator?
I think if you're in Experian or Expedia world [recent data breaches], it could be a differentiator, although I guess they can do whatever they want.
Steven Shack:  I mean, how much does a congressional hearing cost you?
Scott Cohen: Yeah, that's what I'm joking about. They can do whatever they want. But, no, seriously, this is a great question. 
Lovan Chetty: But I think especially with some of the new regulations that are coming predominantly in Europe, there's very explicit monetary amounts that someone's going to have to pay for making these mistakes. It goes beyond reputation now, because there's extra legislation that's starting to come in.
Scott Cohen:  The EU is being pretty heavy handed with this, and that is another great way of forcing adoption.
Steven Shack: I think it's like safety was for the automakers in the 60's. It's looked at as a cost center, "Oh, I gotta add fancy brakes and seat belts and stuff," but very quickly, some companies can use that as a competitive advantage and as an actual feature point in their product.
Scott Shagory:
I would certainly agree with that. When I've spoken with folks, particularly in the startup space, where they're attacking a particular industry or looking at it for midsize companies, et cetera, they definitely notice a demarcation between some firms, and frankly some industry sectors that, don't really care about cybersecurity at its core. It's really either about compliance or about mitigation, in some particular way, and so they do. They view it as a cost center, just like Steven mentioned.
There are some others, however, who have started thinking about it instead as a strategic asset and whether or not their customers really value some sort of additional cyber infrastructure, either in a product or in a service or however that might be defined. But the reality is still frustrating I’m sure to a lot of cyber professionals, which is that there are sectors that really just don't care about it or don't want to think about it or won't pay for it.
I mean, IOT, that I spend time talking with people about, particularly in the consumer space, is an easy example. But there are plenty of others where you would think either they should care or where the conversation is difficult to have, such as in healthcare, where for many healthcare professionals, their overriding training is first "Do no harm," and they're really thinking about patient health and wellbeing -- and not so much per se about cybersecurity as an entity in its own. 
The conversation ends up being more interesting as you try and think about ‘what's your go-to market strategy?’  As Steven said, could it [security] be part of the feature set?  I think clearly the answer is yes.
But for quite a few different industries, it's really a challenge to try to figure out what that is. Or if you're just a smaller participant, and you don't have a huge budget, what does cybersecurity really mean for you internally and then in your product and your portfolio and partnerships? It's a great question, actually.
Scott Cohen: If only there were a consulting firm that could help with that dilemma! That is my last shameless plug for And we are out of time. So, thanks everyone. Thank you, Scott. Thank you, Steven. And thank you, Lovan. I enjoyed it. Good topics. Hope to see you on another round table. 
Scott Shagory: Thanks so much
Scott Cohen:  Take care, everyone.
Lovan Chetty Cheers, everyone.
Steven Shack: Bye now.