1 00:00:02,000 --> 00:00:03,319 hello 2 00:00:03,319 --> 00:00:06,839 everyone we are getting started here on 3 00:00:06,839 --> 00:00:09,880 our August lunch and learn session 4 00:00:09,880 --> 00:00:12,639 presented by Ken group's Atlas customer 5 00:00:12,639 --> 00:00:16,400 experience team my name is Alice Deane I 6 00:00:16,400 --> 00:00:19,160 am the engineering manager for the atlas 7 00:00:19,160 --> 00:00:21,960 customer experience team and I'm excited 8 00:00:21,960 --> 00:00:24,800 to be presenting this month's session on 9 00:00:24,800 --> 00:00:28,000 intermediate level Splunk searching so 10 00:00:28,000 --> 00:00:30,199 thank you all for attending I hope you 11 00:00:30,199 --> 00:00:33,000 get some good ideas out of this uh and 12 00:00:33,000 --> 00:00:35,120 certainly encourage engagement through 13 00:00:35,120 --> 00:00:37,040 the chat uh and I'll have some 14 00:00:37,040 --> 00:00:39,800 information at the end on following up 15 00:00:39,800 --> 00:00:42,239 uh and speaking with my team directly on 16 00:00:42,239 --> 00:00:45,879 any issues uh or interests that you have 17 00:00:45,879 --> 00:00:48,000 uh around these types of Concepts that 18 00:00:48,000 --> 00:00:51,520 we're going to cover today so uh jumping 19 00:00:51,520 --> 00:00:55,199 into an intermediate level uh session I 20 00:00:55,199 --> 00:00:57,960 I do want to say that we have previously 21 00:00:57,960 --> 00:01:02,120 done a basic level uh searching uh 22 00:01:02,120 --> 00:01:05,280 session so that we are really 23 00:01:05,280 --> 00:01:07,360 progressing from that picking up right 24 00:01:07,360 --> 00:01:09,400 where we left off uh we've done that 25 00:01:09,400 --> 00:01:10,640 session with quite a few of our 26 00:01:10,640 --> 00:01:12,920 customers individually uh and highly 27 00:01:12,920 --> 00:01:14,640 recommend if you're interested in doing 28 00:01:14,640 --> 00:01:18,200 that or this session with a larger team 29 00:01:18,200 --> 00:01:19,920 uh we're happy to to discuss and 30 00:01:19,920 --> 00:01:22,840 coordinate that uh so getting started 31 00:01:22,840 --> 00:01:25,600 we're going to take a look at the final 32 00:01:25,600 --> 00:01:29,000 search uh from our basic search session 33 00:01:29,000 --> 00:01:31,159 uh and we're going to walk through that 34 00:01:31,159 --> 00:01:34,159 understand some of the concepts uh and 35 00:01:34,159 --> 00:01:36,479 then we're going to take a step back 36 00:01:36,479 --> 00:01:39,479 look a little more generally at SPL 37 00:01:39,479 --> 00:01:41,759 operations and understanding how 38 00:01:41,759 --> 00:01:46,200 different commands apply to data uh and 39 00:01:46,200 --> 00:01:49,320 really that next level of understanding 40 00:01:49,320 --> 00:01:51,759 for how you can write more complex 41 00:01:51,759 --> 00:01:54,119 searches and understand really uh when 42 00:01:54,119 --> 00:01:57,119 to use certain types of commands uh and 43 00:01:57,119 --> 00:01:59,560 of course uh in the session we're going 44 00:01:59,560 --> 00:02:04,399 to to have a uh series of demos uh using 45 00:02:04,399 --> 00:02:07,360 a few specific commands highlighting the 46 00:02:07,360 --> 00:02:10,440 different SPL command types uh that we 47 00:02:10,440 --> 00:02:12,840 discuss in the second portion uh and get 48 00:02:12,840 --> 00:02:15,879 to see that on the tutorial data uh that 49 00:02:15,879 --> 00:02:18,160 you can also use uh in your environment 50 00:02:18,160 --> 00:02:20,840 in a test environment uh very 51 00:02:20,840 --> 00:02:24,200 simply so I will always encourage uh 52 00:02:24,200 --> 00:02:27,720 especially with search content that you 53 00:02:27,720 --> 00:02:30,319 look into the additional resource that I 54 00:02:30,319 --> 00:02:34,120 have listed here the uh search reference 55 00:02:34,120 --> 00:02:36,440 documentation is one of my favorite 56 00:02:36,440 --> 00:02:38,760 bookmarks that I use frequently in my 57 00:02:38,760 --> 00:02:41,000 own environments and working in customer 58 00:02:41,000 --> 00:02:43,560 environments uh it is really the the 59 00:02:43,560 --> 00:02:46,000 best quick resource to get information 60 00:02:46,000 --> 00:02:49,560 on syntax and examples of any Search 61 00:02:49,560 --> 00:02:51,760 Command uh and is always a great 62 00:02:51,760 --> 00:02:55,000 resource to have the search manual is a 63 00:02:55,000 --> 00:02:57,080 little bit more conceptual but as you're 64 00:02:57,080 --> 00:02:59,120 learning more about different types of 65 00:02:59,120 --> 00:03:00,360 search operations 66 00:03:00,360 --> 00:03:02,440 it's very helpful to be able to review 67 00:03:02,440 --> 00:03:03,360 this 68 00:03:03,360 --> 00:03:05,560 documentation and have reference 69 00:03:05,560 --> 00:03:08,680 material that you can come back to uh as 70 00:03:08,680 --> 00:03:11,080 you are studying and trying to get 71 00:03:11,080 --> 00:03:13,480 better and writing more complex uh 72 00:03:13,480 --> 00:03:16,879 search content I have also linked here 73 00:03:16,879 --> 00:03:18,959 the documentation on how to use the 74 00:03:18,959 --> 00:03:21,799 Splunk tutorial data uh so if you've not 75 00:03:21,799 --> 00:03:23,360 done that before it's a very simple 76 00:03:23,360 --> 00:03:25,920 process uh and there are consistently 77 00:03:25,920 --> 00:03:28,280 updated download files that spun 78 00:03:28,280 --> 00:03:30,680 provides uh that you're able to directly 79 00:03:30,680 --> 00:03:33,439 upload into any spunk environment so 80 00:03:33,439 --> 00:03:35,560 that's what I'm going to be using today 81 00:03:35,560 --> 00:03:39,000 uh and given that you are searching over 82 00:03:39,000 --> 00:03:41,400 uh appropriate time windows for when you 83 00:03:41,400 --> 00:03:43,920 download the tutorial data set uh these 84 00:03:43,920 --> 00:03:46,519 searches will work uh on the tutorial 85 00:03:46,519 --> 00:03:48,760 data as well so highly encourage after 86 00:03:48,760 --> 00:03:50,879 the fact if you want to go through uh 87 00:03:50,879 --> 00:03:53,760 and test out some of the content um 88 00:03:53,760 --> 00:03:56,920 you'll be able to access a recording as 89 00:03:56,920 --> 00:03:59,360 well as if you'd like the slides that 90 00:03:59,360 --> 00:04:00,959 I'm Pres presenting off of today which I 91 00:04:00,959 --> 00:04:02,280 highly encourage because there are a lot 92 00:04:02,280 --> 00:04:04,799 of useful links in here uh reach out to 93 00:04:04,799 --> 00:04:06,760 my team again right at the end of the 94 00:04:06,760 --> 00:04:08,599 slides we'll have that 95 00:04:08,599 --> 00:04:13,079 info so looking at our overview of basic 96 00:04:13,079 --> 00:04:15,799 search uh I just want to cover 97 00:04:15,799 --> 00:04:18,120 conceptually uh the two categories that 98 00:04:18,120 --> 00:04:21,639 we discuss in that session and so those 99 00:04:21,639 --> 00:04:24,199 two are the statistical and charting 100 00:04:24,199 --> 00:04:28,479 functions uh which consist of in those 101 00:04:28,479 --> 00:04:31,479 demos aggregate and time functions so 102 00:04:31,479 --> 00:04:33,919 aggregate functions are going to be your 103 00:04:33,919 --> 00:04:37,400 commonly used statistical functions uh 104 00:04:37,400 --> 00:04:40,400 meant for summarization uh and then time 105 00:04:40,400 --> 00:04:43,199 functions actually using uh the 106 00:04:43,199 --> 00:04:46,639 timestamp field underscore time or any 107 00:04:46,639 --> 00:04:48,600 other time that you've extracted from 108 00:04:48,600 --> 00:04:51,759 data uh and looking at earliest latest 109 00:04:51,759 --> 00:04:55,000 uh relative time values uh in a 110 00:04:55,000 --> 00:04:58,240 summative fashion and then evaluation 111 00:04:58,240 --> 00:05:02,320 functions are the separate uh type where 112 00:05:02,320 --> 00:05:04,400 we discuss comparison and conditional 113 00:05:04,400 --> 00:05:07,600 statements so using your if and your 114 00:05:07,600 --> 00:05:10,600 case commands uh in 115 00:05:10,600 --> 00:05:14,120 evals uh also datetime functions that 116 00:05:14,120 --> 00:05:17,160 apply operations to events uniquely uh 117 00:05:17,160 --> 00:05:19,759 so not necessarily summarization but 118 00:05:19,759 --> 00:05:22,280 interacting with the time values 119 00:05:22,280 --> 00:05:24,319 themselves maybe changing the time 120 00:05:24,319 --> 00:05:27,000 format uh and then multivalue evalve 121 00:05:27,000 --> 00:05:29,360 functions we touch on that very lightly 122 00:05:29,360 --> 00:05:31,720 uh and it is more conceptual in basic 123 00:05:31,720 --> 00:05:34,000 search So today we're going to dive in 124 00:05:34,000 --> 00:05:36,120 as part of our demo and look at 125 00:05:36,120 --> 00:05:39,160 multivalue eval functions uh later in 126 00:05:39,160 --> 00:05:41,319 the 127 00:05:41,479 --> 00:05:44,880 presentation so on this slide here I 128 00:05:44,880 --> 00:05:48,800 have highlighted uh in Gray the search 129 00:05:48,800 --> 00:05:52,120 that we end basic search with uh and so 130 00:05:52,120 --> 00:05:55,000 that is broken up into three segments 131 00:05:55,000 --> 00:05:57,479 where we have uh the first line being a 132 00:05:57,479 --> 00:06:00,240 filter to a data set uh this is very 133 00:06:00,240 --> 00:06:03,120 simply how you are sourcing most of your 134 00:06:03,120 --> 00:06:06,319 data in most of your searches in Splunk 135 00:06:06,319 --> 00:06:08,000 uh and we always want to be a specific 136 00:06:08,000 --> 00:06:11,000 as possible you'll most often see the 137 00:06:11,000 --> 00:06:13,039 The Logical way to do that is by 138 00:06:13,039 --> 00:06:15,680 identifying an index and a source type 139 00:06:15,680 --> 00:06:18,120 possibly some specific values of given 140 00:06:18,120 --> 00:06:20,199 fields in that data before you start 141 00:06:20,199 --> 00:06:22,720 applying other operations in our case we 142 00:06:22,720 --> 00:06:25,199 want to work with a whole data set uh 143 00:06:25,199 --> 00:06:28,880 and then we mve into applying our eval 144 00:06:28,880 --> 00:06:30,120 statements 145 00:06:30,120 --> 00:06:33,080 so in the evals the purpose of these is 146 00:06:33,080 --> 00:06:36,560 to create some new fields to work with 147 00:06:36,560 --> 00:06:40,080 uh and so we have two operations here uh 148 00:06:40,080 --> 00:06:42,440 and you can see that on the first line 149 00:06:42,440 --> 00:06:46,120 we're starting with an error check field 150 00:06:46,120 --> 00:06:49,160 uh these are web access logs so we're 151 00:06:49,160 --> 00:06:52,720 looking at the HTTP status codes as the 152 00:06:52,720 --> 00:06:56,039 status field and we have a logical 153 00:06:56,039 --> 00:06:57,599 condition here we greater than or equal 154 00:06:57,599 --> 00:07:00,680 to 400 we want to return errors and so 155 00:07:00,680 --> 00:07:04,120 very simple example uh making it as easy 156 00:07:04,120 --> 00:07:05,879 as possible if you want to get specifics 157 00:07:05,879 --> 00:07:08,720 on your 200s and your 300s it's the 158 00:07:08,720 --> 00:07:11,639 exact same type of logic to go and apply 159 00:07:11,639 --> 00:07:14,120 uh likely a case statement to get some 160 00:07:14,120 --> 00:07:17,199 additional conditions uh and more unique 161 00:07:17,199 --> 00:07:20,520 output in an error check or some sort of 162 00:07:20,520 --> 00:07:23,800 field uh indicating uh what you want to 163 00:07:23,800 --> 00:07:25,919 see out of your status code so this case 164 00:07:25,919 --> 00:07:30,080 simple errors or the value of non here 165 00:07:30,080 --> 00:07:32,120 if we have say at 166 00:07:32,120 --> 00:07:35,400 200 we're also using a Time function to 167 00:07:35,400 --> 00:07:39,160 create a second field called day uh you 168 00:07:39,160 --> 00:07:41,759 may be familiar with some of the uh 169 00:07:41,759 --> 00:07:46,360 fields that you get out of uh by default 170 00:07:46,360 --> 00:07:49,759 for most any events in Splunk uh and 171 00:07:49,759 --> 00:07:51,759 that they're related to breakdowns of 172 00:07:51,759 --> 00:07:56,000 the time stamps uh you have day month uh 173 00:07:56,000 --> 00:07:58,240 and many others in this case I want to 174 00:07:58,240 --> 00:08:00,560 get a specific format for day so we use 175 00:08:00,560 --> 00:08:03,479 a strif Time function uh and we have a 176 00:08:03,479 --> 00:08:07,039 time format variable here on the actual 177 00:08:07,039 --> 00:08:10,280 extracted time stamp first of BL so 178 00:08:10,280 --> 00:08:12,039 coming out of the second line we've 179 00:08:12,039 --> 00:08:14,319 accessed our data we have created two 180 00:08:14,319 --> 00:08:17,479 new fields to use and then we are 181 00:08:17,479 --> 00:08:20,960 actually performing uh charting with a 182 00:08:20,960 --> 00:08:23,680 statistical function and so that is 183 00:08:23,680 --> 00:08:26,240 using time chart and we can see here 184 00:08:26,240 --> 00:08:29,159 that we are counting our events that 185 00:08:29,159 --> 00:08:33,479 actually have the error value uh for our 186 00:08:33,479 --> 00:08:36,000 created error check field and so I'm 187 00:08:36,000 --> 00:08:39,279 going to Pivot over to uh Splunk here 188 00:08:39,279 --> 00:08:40,880 and we're going to look at this search 189 00:08:40,880 --> 00:08:43,440 and I have commented out uh most of the 190 00:08:43,440 --> 00:08:46,279 logic we'll step back through it uh we 191 00:08:46,279 --> 00:08:49,200 are looking at our web access log events 192 00:08:49,200 --> 00:08:52,800 here uh and we want to then apply our 193 00:08:52,800 --> 00:08:58,240 eval and so by applying the eval we can 194 00:08:58,240 --> 00:09:01,279 get our error check field that provides 195 00:09:01,279 --> 00:09:03,279 error or non-error we're seeing that we 196 00:09:03,279 --> 00:09:05,160 have mostly non-error 197 00:09:05,160 --> 00:09:09,680 events uh and then we have the day field 198 00:09:09,680 --> 00:09:11,760 and so day is actually providing the 199 00:09:11,760 --> 00:09:14,440 full name of day for the time stamp for 200 00:09:14,440 --> 00:09:17,800 all these events so with our time chart 201 00:09:17,800 --> 00:09:22,200 this is the summarization uh with a 202 00:09:22,200 --> 00:09:24,160 condition actually that we're spanning 203 00:09:24,160 --> 00:09:27,720 by default over a single day so this may 204 00:09:27,720 --> 00:09:31,839 not be a very logical use of a split by 205 00:09:31,839 --> 00:09:34,360 day when we are already using a time 206 00:09:34,360 --> 00:09:37,079 chart command that is dividing our 207 00:09:37,079 --> 00:09:41,040 results by the time bin uh effectively a 208 00:09:41,040 --> 00:09:46,079 span of one day but what we can do uh is 209 00:09:46,079 --> 00:09:50,440 change our split by field to host and 210 00:09:50,440 --> 00:09:52,600 get a little bit more of a reasonable 211 00:09:52,600 --> 00:09:54,720 presentation we were able to see with 212 00:09:54,720 --> 00:09:57,720 the counts in the individual days not 213 00:09:57,720 --> 00:09:59,600 only split through the time chart but by 214 00:09:59,600 --> 00:10:02,399 the day field that we only had values 215 00:10:02,399 --> 00:10:04,959 where our Matrix matched up for the 216 00:10:04,959 --> 00:10:09,680 actual day so here we have our uh hosts 217 00:10:09,680 --> 00:10:12,640 one two and three and then across days 218 00:10:12,640 --> 00:10:15,640 counts of the error events that we 219 00:10:15,640 --> 00:10:20,160 observe so that is uh the search that we 220 00:10:20,160 --> 00:10:22,440 end on in basic search the concepts 221 00:10:22,440 --> 00:10:25,040 there being accessing our data uh 222 00:10:25,040 --> 00:10:27,279 searching in a descriptive manner using 223 00:10:27,279 --> 00:10:29,320 our metadata Fields the index and the 224 00:10:29,320 --> 00:10:32,200 Source type uh the evaluation functions 225 00:10:32,200 --> 00:10:33,920 where we're creating new Fields 226 00:10:33,920 --> 00:10:37,639 manipulating data uh and then we have a 227 00:10:37,639 --> 00:10:40,200 time chart function uh that is providing 228 00:10:40,200 --> 00:10:42,880 some uh summarized statistics here based 229 00:10:42,880 --> 00:10:44,480 on the time 230 00:10:44,480 --> 00:10:48,680 range so we will pivot back and we're 231 00:10:48,680 --> 00:10:51,399 going to take a step back out of the SPL 232 00:10:51,399 --> 00:10:54,200 for a second just to talk about these 233 00:10:54,200 --> 00:10:56,519 different kinds of search operations 234 00:10:56,519 --> 00:10:59,360 that we just performed so you'll hear 235 00:10:59,360 --> 00:11:03,079 these terms uh if you are really kind of 236 00:11:03,079 --> 00:11:06,040 diving deeper into actual operations of 237 00:11:06,040 --> 00:11:09,920 Splunk searching uh and you can get very 238 00:11:09,920 --> 00:11:12,560 detailed regarding the optimization of 239 00:11:12,560 --> 00:11:16,279 searches uh around uh these types of 240 00:11:16,279 --> 00:11:17,680 commands and the order in which you 241 00:11:17,680 --> 00:11:21,399 choose to execute SPL today I'm going to 242 00:11:21,399 --> 00:11:24,240 focus on how these operations actually 243 00:11:24,240 --> 00:11:27,240 apply to the data and helping you to 244 00:11:27,240 --> 00:11:29,320 make better decisions about what 245 00:11:29,320 --> 00:11:32,320 commands are best for the scenario that 246 00:11:32,320 --> 00:11:34,240 you have or the output that you want to 247 00:11:34,240 --> 00:11:37,639 see uh and in future sessions we will 248 00:11:37,639 --> 00:11:39,360 discuss the actual optimization of 249 00:11:39,360 --> 00:11:42,079 searches uh through this optimal order 250 00:11:42,079 --> 00:11:46,440 of functions uh and some other means uh 251 00:11:46,440 --> 00:11:48,200 but just a caveat there that we're going 252 00:11:48,200 --> 00:11:50,440 to talk uh pretty specifically today 253 00:11:50,440 --> 00:11:52,839 just about these uh individually how 254 00:11:52,839 --> 00:11:54,720 they work with data uh and then how you 255 00:11:54,720 --> 00:11:55,839 see them in 256 00:11:55,839 --> 00:11:59,839 combination so our types of SPL command 257 00:11:59,839 --> 00:12:03,160 the top three in bold we'll focus on in 258 00:12:03,160 --> 00:12:06,079 our examples the first of which is 259 00:12:06,079 --> 00:12:07,279 streaming 260 00:12:07,279 --> 00:12:10,760 operations uh which are executed on 261 00:12:10,760 --> 00:12:13,079 individual events as they return by a 262 00:12:13,079 --> 00:12:15,399 search uh so you can think of this like 263 00:12:15,399 --> 00:12:16,120 your 264 00:12:16,120 --> 00:12:18,880 eval um that is going to be doing 265 00:12:18,880 --> 00:12:21,440 something to every single event uh 266 00:12:21,440 --> 00:12:24,279 modifying Fields when they're available 267 00:12:24,279 --> 00:12:28,399 uh we do have generating functions uh so 268 00:12:28,399 --> 00:12:30,800 generating function are going to be used 269 00:12:30,800 --> 00:12:33,839 situationally where you're sourcing data 270 00:12:33,839 --> 00:12:38,079 from uh non-indexed data sets and so you 271 00:12:38,079 --> 00:12:40,839 would see that from uh either input 272 00:12:40,839 --> 00:12:43,760 lookup commands uh or maybe tstats 273 00:12:43,760 --> 00:12:46,120 pulling information from the tsid X 274 00:12:46,120 --> 00:12:48,920 Files uh and so generating the 275 00:12:48,920 --> 00:12:51,079 statistical output based on the data 276 00:12:51,079 --> 00:12:55,040 available there transforming commands uh 277 00:12:55,040 --> 00:12:58,560 you will see uh as often as streaming 278 00:12:58,560 --> 00:13:00,600 commands generally speaking and more 279 00:13:00,600 --> 00:13:02,800 often than generating commands where 280 00:13:02,800 --> 00:13:05,399 transforming is intended to order 281 00:13:05,399 --> 00:13:08,519 results into a data table and I often 282 00:13:08,519 --> 00:13:11,320 think of this much like how we discuss 283 00:13:11,320 --> 00:13:13,639 the statistical functions in basic 284 00:13:13,639 --> 00:13:17,160 search as summarization functions where 285 00:13:17,160 --> 00:13:19,519 you're looking to condense your overall 286 00:13:19,519 --> 00:13:22,680 data set uh into really manageable 287 00:13:22,680 --> 00:13:24,880 consumable results uh so these 288 00:13:24,880 --> 00:13:28,320 operations that apply that summarization 289 00:13:28,320 --> 00:13:31,720 are transform perform we do have two 290 00:13:31,720 --> 00:13:35,600 additional types of SPL commands uh the 291 00:13:35,600 --> 00:13:39,480 first is orchestrating uh you can read 292 00:13:39,480 --> 00:13:41,680 about these I will not discuss in great 293 00:13:41,680 --> 00:13:45,199 detail uh they are used to manipulate 294 00:13:45,199 --> 00:13:48,639 how searches are actually U processed or 295 00:13:48,639 --> 00:13:50,800 or how commands are processed uh and 296 00:13:50,800 --> 00:13:54,079 they don't directly affect the results 297 00:13:54,079 --> 00:13:56,079 in a search how we think about say 298 00:13:56,079 --> 00:13:59,839 applying a stats or an eval uh to a data 299 00:13:59,839 --> 00:14:02,320 set uh so if you're interested 300 00:14:02,320 --> 00:14:04,399 definitely check it out uh link 301 00:14:04,399 --> 00:14:07,720 documentation has details there um data 302 00:14:07,720 --> 00:14:11,120 set processing is seen much more often 303 00:14:11,120 --> 00:14:15,000 uh and you do have uh some conditional 304 00:14:15,000 --> 00:14:18,680 uh scenarios where commands can act as 305 00:14:18,680 --> 00:14:21,759 data set processing so the uh 306 00:14:21,759 --> 00:14:23,959 distinction for data set processing is 307 00:14:23,959 --> 00:14:26,360 going to be that you are operating in 308 00:14:26,360 --> 00:14:29,800 bulk on a single completed data set at 309 00:14:29,800 --> 00:14:32,240 one time so we'll we'll look at an 310 00:14:32,240 --> 00:14:33,920 example of 311 00:14:33,920 --> 00:14:36,600 that I want to Pivot back to our main 312 00:14:36,600 --> 00:14:38,360 three that we're going to be focusing on 313 00:14:38,360 --> 00:14:39,839 and I have mentioned some of these 314 00:14:39,839 --> 00:14:43,800 examples already uh the eval functions 315 00:14:43,800 --> 00:14:45,880 that we've been talking about so far are 316 00:14:45,880 --> 00:14:47,920 perfect examples of our streaming 317 00:14:47,920 --> 00:14:51,440 commands uh so where we are creating new 318 00:14:51,440 --> 00:14:55,600 fields for each entry or log event uh 319 00:14:55,600 --> 00:14:59,399 where we are modifying values for all of 320 00:14:59,399 --> 00:15:01,920 the results that are available uh that 321 00:15:01,920 --> 00:15:05,279 is where we are streaming um with the 322 00:15:05,279 --> 00:15:08,560 search functions input lookup is 323 00:15:08,560 --> 00:15:09,959 possibly one of the most common 324 00:15:09,959 --> 00:15:12,399 generating commands that I see uh 325 00:15:12,399 --> 00:15:15,199 because someone is intending to uh 326 00:15:15,199 --> 00:15:18,720 Source a data set stored in a CSV file 327 00:15:18,720 --> 00:15:21,480 or a KV store collection uh and you're 328 00:15:21,480 --> 00:15:23,720 able to bring that back as a report and 329 00:15:23,720 --> 00:15:26,959 use that logic uh in your 330 00:15:26,959 --> 00:15:29,639 queries so that is 331 00:15:29,639 --> 00:15:33,399 uh not requiring the index data uh or 332 00:15:33,399 --> 00:15:35,560 any index data to actually return the 333 00:15:35,560 --> 00:15:38,120 results that you want to 334 00:15:38,120 --> 00:15:41,319 see and we've talked about stats very 335 00:15:41,319 --> 00:15:43,600 generally speaking uh with a lot of 336 00:15:43,600 --> 00:15:46,440 unique functions you can apply there uh 337 00:15:46,440 --> 00:15:49,560 where this is going to provide a tabular 338 00:15:49,560 --> 00:15:53,560 output uh and is serving that purpose of 339 00:15:53,560 --> 00:15:54,800 summarization so we're really 340 00:15:54,800 --> 00:15:57,560 reformatting the data uh into that 341 00:15:57,560 --> 00:16:00,920 tabular report 342 00:16:02,000 --> 00:16:06,519 so we see in this example search here uh 343 00:16:06,519 --> 00:16:09,000 that we are often combining these 344 00:16:09,000 --> 00:16:12,360 different types of search operations so 345 00:16:12,360 --> 00:16:15,240 in this example that we have uh I have 346 00:16:15,240 --> 00:16:19,319 data that already exists in a CSV file 347 00:16:19,319 --> 00:16:22,839 we are applying a streaming command here 348 00:16:22,839 --> 00:16:26,000 uh where evaluating each line to see if 349 00:16:26,000 --> 00:16:28,399 we match a condition and then returning 350 00:16:28,399 --> 00:16:29,639 the results 351 00:16:29,639 --> 00:16:32,240 based on that evaluation and then we're 352 00:16:32,240 --> 00:16:34,199 applying a transforming command at the 353 00:16:34,199 --> 00:16:36,639 end which is that stats summarization 354 00:16:36,639 --> 00:16:40,480 getting the maximum values uh for the uh 355 00:16:40,480 --> 00:16:44,319 count of errors and the host that is 356 00:16:44,319 --> 00:16:47,600 associated with that so let's PIV over 357 00:16:47,600 --> 00:16:52,079 to Splunk and we'll take a look at that 358 00:16:54,160 --> 00:16:56,319 example so I'm just going to grab my 359 00:16:56,319 --> 00:16:59,440 search here and I pre- commented out 360 00:16:59,440 --> 00:17:03,519 uh the specific uh lines following input 361 00:17:03,519 --> 00:17:06,079 lookup just to see that this generating 362 00:17:06,079 --> 00:17:07,799 command here is not looking for any 363 00:17:07,799 --> 00:17:10,160 specific index data uh we're pulling 364 00:17:10,160 --> 00:17:13,240 directly the results that I have in a 365 00:17:13,240 --> 00:17:17,720 CSV file uh here into this output and so 366 00:17:17,720 --> 00:17:20,520 we have a count of Errors observed 367 00:17:20,520 --> 00:17:25,439 across multiple hosts our where command 368 00:17:25,439 --> 00:17:28,520 uh you might think is reformatting data 369 00:17:28,520 --> 00:17:31,000 in this sense it it is transforming the 370 00:17:31,000 --> 00:17:34,160 results but the evaluation of a wear 371 00:17:34,160 --> 00:17:37,320 function does apply effectively to every 372 00:17:37,320 --> 00:17:41,760 event that is returned uh so it is u a 373 00:17:41,760 --> 00:17:43,960 streaming command that is going to 374 00:17:43,960 --> 00:17:46,559 filter down our result set based on our 375 00:17:46,559 --> 00:17:49,120 condition that the error count is less 376 00:17:49,120 --> 00:17:50,919 than 377 00:17:50,919 --> 00:17:54,760 200 so the following line is our 378 00:17:54,760 --> 00:17:57,320 transforming command where we have two 379 00:17:57,320 --> 00:18:02,240 results left uh 187 for host 3 we want 380 00:18:02,240 --> 00:18:06,039 to see our maximum values here of 187 on 381 00:18:06,039 --> 00:18:09,960 host 3 so our scenario here has really 382 00:18:09,960 --> 00:18:13,400 uh covered where you may have uh hosts 383 00:18:13,400 --> 00:18:15,960 that are trending toward a negative 384 00:18:15,960 --> 00:18:19,280 State you're aware that uh the second 385 00:18:19,280 --> 00:18:22,039 host had already exceeded its uh 386 00:18:22,039 --> 00:18:25,360 threshold value for errors but host 3 387 00:18:25,360 --> 00:18:27,440 also appears to be trending toward this 388 00:18:27,440 --> 00:18:30,159 threshold uh so being able to combine 389 00:18:30,159 --> 00:18:33,000 these types of commands uh understand 390 00:18:33,000 --> 00:18:35,240 the logical condition that you're 391 00:18:35,240 --> 00:18:37,679 searching for uh and then also providing 392 00:18:37,679 --> 00:18:40,840 that consumable output uh so combining 393 00:18:40,840 --> 00:18:44,480 all three of our types of commands 394 00:18:45,320 --> 00:18:49,440 here so uh I'm going to jump to an SPL 395 00:18:49,440 --> 00:18:53,159 demo and as I go through these different 396 00:18:53,159 --> 00:18:55,840 commands uh I'm going to be referencing 397 00:18:55,840 --> 00:18:58,360 back to the different command types that 398 00:18:58,360 --> 00:19:00,080 we're working with I'm going to 399 00:19:00,080 --> 00:19:02,360 introduce in a lot of these searches uh 400 00:19:02,360 --> 00:19:04,679 a lot of small commands uh that I won't 401 00:19:04,679 --> 00:19:07,000 talk about in great detail and that 402 00:19:07,000 --> 00:19:09,360 really is the purpose of using your 403 00:19:09,360 --> 00:19:11,640 search manual uh using your search 404 00:19:11,640 --> 00:19:14,760 reference documentation uh so I will 405 00:19:14,760 --> 00:19:17,400 glance over the use case uh talk about 406 00:19:17,400 --> 00:19:19,559 how it's meant to be applied and then 407 00:19:19,559 --> 00:19:22,200 using in your own scenarios uh where you 408 00:19:22,200 --> 00:19:24,400 have problem you need to solve uh 409 00:19:24,400 --> 00:19:26,880 referencing the docs to find out where 410 00:19:26,880 --> 00:19:29,960 you can apply uh similar functions to 411 00:19:29,960 --> 00:19:32,559 what we observe in the the demonstration 412 00:19:32,559 --> 00:19:36,760 here so the First Command I'm going to 413 00:19:36,760 --> 00:19:40,880 focus on is the Rex command so Rex is a 414 00:19:40,880 --> 00:19:43,480 streaming command that you often see 415 00:19:43,480 --> 00:19:46,559 applied to data sets that do not fully 416 00:19:46,559 --> 00:19:49,720 have data extracted in the format that 417 00:19:49,720 --> 00:19:53,159 you want to be using um in your 418 00:19:53,159 --> 00:19:56,760 reporting or in your logic uh and so 419 00:19:56,760 --> 00:20:00,120 this could very well be handled actually 420 00:20:00,120 --> 00:20:03,440 in the uh configuration of props and 421 00:20:03,440 --> 00:20:06,080 transforms and extracting fields at the 422 00:20:06,080 --> 00:20:08,480 right times and indexing data but as 423 00:20:08,480 --> 00:20:10,280 your bringing new data sources you need 424 00:20:10,280 --> 00:20:12,480 to understand what's available for use 425 00:20:12,480 --> 00:20:14,360 in spunk a lot of times you'll find 426 00:20:14,360 --> 00:20:16,840 yourself needing to extract new fields 427 00:20:16,840 --> 00:20:19,200 in line in your searches uh and be able 428 00:20:19,200 --> 00:20:22,080 to use those in your search Logic Rex 429 00:20:22,080 --> 00:20:28,039 also has uh a said mode that I also see 430 00:20:28,039 --> 00:20:31,600 testing done for masking of data in line 431 00:20:31,600 --> 00:20:34,080 prior to actually putting that into 432 00:20:34,080 --> 00:20:35,120 indexing 433 00:20:35,120 --> 00:20:38,000 configurations um so Rex you would 434 00:20:38,000 --> 00:20:41,200 generally see used um when you don't 435 00:20:41,200 --> 00:20:43,039 have those fields available you need to 436 00:20:43,039 --> 00:20:45,640 use them at that time uh and then we're 437 00:20:45,640 --> 00:20:47,120 going to take a look at an example of 438 00:20:47,120 --> 00:20:49,640 masking data as well uh to test your 439 00:20:49,640 --> 00:20:53,480 Syntax for a said style replace uh in 440 00:20:53,480 --> 00:21:00,600 config files so we will jump back over 441 00:21:04,679 --> 00:21:06,880 so I'm going to start with a search on 442 00:21:06,880 --> 00:21:10,120 an index Source type uh my tutorial data 443 00:21:10,120 --> 00:21:13,159 and then this is actual uh Linux secure 444 00:21:13,159 --> 00:21:16,159 logging uh so these are going to be OS 445 00:21:16,159 --> 00:21:19,039 security logs and we're looking at all 446 00:21:19,039 --> 00:21:21,039 of our web hosts uh that we've been 447 00:21:21,039 --> 00:21:22,440 focusing on 448 00:21:22,440 --> 00:21:25,000 previously in our events you can see 449 00:21:25,000 --> 00:21:29,039 that we have uh first here uh an EV that 450 00:21:29,039 --> 00:21:31,720 has fail password for invalid user in 451 00:21:31,720 --> 00:21:34,320 that we're provided a source IP a source 452 00:21:34,320 --> 00:21:36,559 port and we go to see the fields that 453 00:21:36,559 --> 00:21:38,919 are extracted and that's that's not 454 00:21:38,919 --> 00:21:41,919 being done for us automatically so just 455 00:21:41,919 --> 00:21:43,880 to start testing our logic to see if we 456 00:21:43,880 --> 00:21:46,799 can get uh the results we want to see 457 00:21:46,799 --> 00:21:49,760 we're going to use the Rex command and 458 00:21:49,760 --> 00:21:53,240 in doing so we are applying this 459 00:21:53,240 --> 00:21:55,440 operation across every event again a 460 00:21:55,440 --> 00:21:59,600 streaming command we are looking at the 461 00:21:59,600 --> 00:22:01,279 raw field so we're actually looking at 462 00:22:01,279 --> 00:22:04,679 the raw text of each of these log events 463 00:22:04,679 --> 00:22:07,480 and then the rec syntax is simply to 464 00:22:07,480 --> 00:22:11,960 provide in double quotes uh a Rex uh 465 00:22:11,960 --> 00:22:14,840 match and we're using named groups for 466 00:22:14,840 --> 00:22:17,440 field extractions so for every single 467 00:22:17,440 --> 00:22:19,440 event that we see failed password for 468 00:22:19,440 --> 00:22:22,919 invalid user we are actually extracting 469 00:22:22,919 --> 00:22:26,400 a user field The Source IP field and the 470 00:22:26,400 --> 00:22:28,799 source Port field for the sake of 471 00:22:28,799 --> 00:22:30,880 Simplicity I tried to keep the RX simple 472 00:22:30,880 --> 00:22:33,760 you can make this as complex as you need 473 00:22:33,760 --> 00:22:37,679 to for your needs for your data uh and 474 00:22:37,679 --> 00:22:40,960 so in our extracted Fields uh I've 475 00:22:40,960 --> 00:22:42,840 actually pre-selected these so we can 476 00:22:42,840 --> 00:22:46,240 see our user is now available and this 477 00:22:46,240 --> 00:22:50,039 applies to the events where the Rex was 478 00:22:50,039 --> 00:22:53,159 actually valid and matching on the uh 479 00:22:53,159 --> 00:22:57,440 failed password for invalid user Etc 480 00:22:57,440 --> 00:23:00,120 string so now that we have our Fields 481 00:23:00,120 --> 00:23:03,799 extracted we can actually use these and 482 00:23:03,799 --> 00:23:04,799 we want 483 00:23:04,799 --> 00:23:09,400 to do a stats count as failed login so 484 00:23:09,400 --> 00:23:13,400 anytime you see uh a an operation as and 485 00:23:13,400 --> 00:23:16,640 then a unique name just a rename uh 486 00:23:16,640 --> 00:23:19,080 through the transformation function 487 00:23:19,080 --> 00:23:21,480 easier way to uh actually keep 488 00:23:21,480 --> 00:23:23,480 consistency uh with referencing your 489 00:23:23,480 --> 00:23:26,760 Fields as well as not have to rename 490 00:23:26,760 --> 00:23:29,919 later on uh with some additional in this 491 00:23:29,919 --> 00:23:31,679 case you'd have to reference the name 492 00:23:31,679 --> 00:23:34,520 distinct count uh so just a way to keep 493 00:23:34,520 --> 00:23:38,320 things clean and easy to use in further 494 00:23:38,320 --> 00:23:42,159 uh lines of SPL so we are counting our 495 00:23:42,159 --> 00:23:43,919 failed logins we're looking at the 496 00:23:43,919 --> 00:23:47,840 distinct count of the source IP values 497 00:23:47,840 --> 00:23:50,000 that we have and then we're splitting 498 00:23:50,000 --> 00:23:52,960 that by the host and the user so you can 499 00:23:52,960 --> 00:23:55,720 see here uh this tutorial data is 500 00:23:55,720 --> 00:23:57,880 actually pretty flat across most of the 501 00:23:57,880 --> 00:24:00,120 sources so we're not going to have uh 502 00:24:00,120 --> 00:24:04,679 any outliers or spikes in our stats here 503 00:24:04,679 --> 00:24:07,960 but you can see the resulting 504 00:24:08,960 --> 00:24:11,440 presentation in line four we do have a 505 00:24:11,440 --> 00:24:14,840 sort command and this is an example of a 506 00:24:14,840 --> 00:24:17,520 data set processing command where we are 507 00:24:17,520 --> 00:24:20,400 actually evaluating a full completed 508 00:24:20,400 --> 00:24:23,640 data set and reordering it uh given the 509 00:24:23,640 --> 00:24:26,000 logic here we want to descend on these 510 00:24:26,000 --> 00:24:29,000 numeric values uh so keep mind as you're 511 00:24:29,000 --> 00:24:31,200 operating on different fields it's going 512 00:24:31,200 --> 00:24:33,799 to be the same sort of either basic 513 00:24:33,799 --> 00:24:37,159 numeric or the lexicographical ordering 514 00:24:37,159 --> 00:24:40,360 that you typically see in 515 00:24:40,840 --> 00:24:45,720 Splunk so we do have a second example uh 516 00:24:45,720 --> 00:24:49,200 with the said style 517 00:24:54,240 --> 00:24:58,640 replace so you can see in my events here 518 00:24:58,640 --> 00:25:01,640 uh we are searching the tutorial and 519 00:25:01,640 --> 00:25:05,039 vendor sales index and Source type and 520 00:25:05,039 --> 00:25:06,720 I've gone ahead and applied one 521 00:25:06,720 --> 00:25:09,399 operation and this is going to be a 522 00:25:09,399 --> 00:25:11,880 helpful operation to understand really 523 00:25:11,880 --> 00:25:14,679 what we are replacing and how to get 524 00:25:14,679 --> 00:25:18,159 consistent operation on these fields uh 525 00:25:18,159 --> 00:25:20,279 so in this case we are actually creating 526 00:25:20,279 --> 00:25:23,559 an ID length field where we are going to 527 00:25:23,559 --> 00:25:26,960 choose to mask the value of account ID 528 00:25:26,960 --> 00:25:29,120 in our Rex command we want to know that 529 00:25:29,120 --> 00:25:31,679 that's a consistent number of characters 530 00:25:31,679 --> 00:25:33,799 uh through all of our data it's very 531 00:25:33,799 --> 00:25:37,080 simple to spot check uh but just to be 532 00:25:37,080 --> 00:25:39,440 certain we want to apply this to all of 533 00:25:39,440 --> 00:25:42,760 our data in this case streaming command 534 00:25:42,760 --> 00:25:45,520 uh through this eval uh we 535 00:25:45,520 --> 00:25:49,279 are uh changing the type of the data 536 00:25:49,279 --> 00:25:51,919 because account ID is actually numeric 537 00:25:51,919 --> 00:25:53,720 we're making that a string value so that 538 00:25:53,720 --> 00:25:56,720 we can look at the length these are 539 00:25:56,720 --> 00:25:58,840 common functions in any programming 540 00:25:58,840 --> 00:26:01,559 languages uh and so the syntax here in 541 00:26:01,559 --> 00:26:04,039 SPL is quite simple uh just to be able 542 00:26:04,039 --> 00:26:06,520 to get that contextual feeli we 543 00:26:06,520 --> 00:26:09,399 understand we have 16 characters for 544 00:26:09,399 --> 00:26:12,480 100% of our events in the account 545 00:26:12,480 --> 00:26:17,000 IDs so actually applying our Rex command 546 00:26:17,000 --> 00:26:20,760 we are going to now specify a unique 547 00:26:20,760 --> 00:26:23,919 field not just uncore raw uh we are 548 00:26:23,919 --> 00:26:27,159 applying the said mode and this is a 549 00:26:27,159 --> 00:26:30,799 said syntax uh replacement uh looking 550 00:26:30,799 --> 00:26:33,559 for the uh it's a capture group for the 551 00:26:33,559 --> 00:26:35,880 first 12 digits uh and then we're 552 00:26:35,880 --> 00:26:39,240 replacing that with a series of 12 X's 553 00:26:39,240 --> 00:26:42,039 so you can see in our first event the 554 00:26:42,039 --> 00:26:45,320 account ID is now masked we only have uh 555 00:26:45,320 --> 00:26:48,520 the remaining four digits to be able to 556 00:26:48,520 --> 00:26:52,320 identify that and so if our data was 557 00:26:52,320 --> 00:26:55,360 indexed and is appropriately done so uh 558 00:26:55,360 --> 00:26:58,039 in Splunk with the full account IDs but 559 00:26:58,039 --> 00:27:00,360 for for the sake of reporting we want to 560 00:27:00,360 --> 00:27:04,840 be able to mask that um for the audience 561 00:27:04,840 --> 00:27:07,799 then we're able to use the the said 562 00:27:07,799 --> 00:27:11,919 replace and then to finalize a report 563 00:27:11,919 --> 00:27:13,880 this is just an example of the top 564 00:27:13,880 --> 00:27:16,399 command which does a few operations 565 00:27:16,399 --> 00:27:18,120 together uh and makes for a good 566 00:27:18,120 --> 00:27:20,720 shorthand report uh taking all the 567 00:27:20,720 --> 00:27:24,080 unique values of the provided field uh 568 00:27:24,080 --> 00:27:26,480 giving you a count of those values and 569 00:27:26,480 --> 00:27:29,000 then showing the percentage 570 00:27:29,000 --> 00:27:31,679 of the makeup for the total data set 571 00:27:31,679 --> 00:27:34,520 that that unique value accounts for so 572 00:27:34,520 --> 00:27:37,399 again pretty flat in this tutorial data 573 00:27:37,399 --> 00:27:40,200 in seeing a very consistent 574 00:27:40,200 --> 00:27:45,159 .3% uh across these different account 575 00:27:46,679 --> 00:27:51,080 IDs so we have looked at a few examples 576 00:27:51,080 --> 00:27:54,640 with the Rex command uh and that is 577 00:27:54,640 --> 00:27:57,039 again streaming we're going to look at 578 00:27:57,039 --> 00:27:59,120 another streaming command 579 00:27:59,120 --> 00:28:02,399 uh which is going to be a set of 580 00:28:02,399 --> 00:28:07,200 multivalue eval functions and so again 581 00:28:07,200 --> 00:28:09,559 if you're to have a bookmark for search 582 00:28:09,559 --> 00:28:12,320 documentation multivalue eval functions 583 00:28:12,320 --> 00:28:14,559 are a great one to have uh because when 584 00:28:14,559 --> 00:28:17,240 you encounter these uh it really takes 585 00:28:17,240 --> 00:28:19,960 some time to figure out how to actually 586 00:28:19,960 --> 00:28:25,960 operate on data um and so the U 587 00:28:25,960 --> 00:28:29,559 multivalue functions are um really just 588 00:28:29,559 --> 00:28:31,799 a collection that depending on your use 589 00:28:31,799 --> 00:28:34,679 case uh you're able to determine the the 590 00:28:34,679 --> 00:28:39,080 best to apply um you see it often used 591 00:28:39,080 --> 00:28:42,840 with uh Json and XML so data formats 592 00:28:42,840 --> 00:28:44,880 that are actually naturally going to 593 00:28:44,880 --> 00:28:47,360 provide uh a multivalue field where you 594 00:28:47,360 --> 00:28:50,480 have repeated tags or Keys uh across 595 00:28:50,480 --> 00:28:54,320 unique uh events as they're extracted uh 596 00:28:54,320 --> 00:28:56,360 and you often see a lot of times in 597 00:28:56,360 --> 00:28:58,480 Windows event logs you actually have 598 00:28:58,480 --> 00:29:01,360 repeated key values uh where your values 599 00:29:01,360 --> 00:29:02,960 are different and the position in the 600 00:29:02,960 --> 00:29:05,200 event is actually specific to a 601 00:29:05,200 --> 00:29:08,840 condition uh so you may have um a need 602 00:29:08,840 --> 00:29:11,440 for extraction or interaction with one 603 00:29:11,440 --> 00:29:14,399 of those unique values uh to actually 604 00:29:14,399 --> 00:29:18,600 get a reasonable outcome from your 605 00:29:18,600 --> 00:29:22,799 data and so um we're going to use 606 00:29:22,799 --> 00:29:25,960 multivalue eval functions uh when we 607 00:29:25,960 --> 00:29:28,679 have a uh change we want to the 608 00:29:28,679 --> 00:29:31,880 presentation of data uh and we're able 609 00:29:31,880 --> 00:29:34,880 to do so with multivalue Fields this I 610 00:29:34,880 --> 00:29:36,720 would say often occurs when you have 611 00:29:36,720 --> 00:29:39,960 multivalue data uh and then you want to 612 00:29:39,960 --> 00:29:43,080 be able to change the the format of the 613 00:29:43,080 --> 00:29:45,640 multivalue fields there uh and then 614 00:29:45,640 --> 00:29:46,960 we're also going to look at a quick 615 00:29:46,960 --> 00:29:51,279 example of uh actually using multivalue 616 00:29:51,279 --> 00:29:54,880 evaluation uh as a logical 617 00:29:54,880 --> 00:30:00,039 condition so uh the first 618 00:30:03,320 --> 00:30:05,679 example we're going to start with a 619 00:30:05,679 --> 00:30:08,720 simple table looking at our web access 620 00:30:08,720 --> 00:30:11,240 logs uh and so we're just going to pull 621 00:30:11,240 --> 00:30:14,880 in our status and refer domain fields 622 00:30:14,880 --> 00:30:18,440 and so you can see uh we've got a uh 623 00:30:18,440 --> 00:30:23,000 HTTP status code uh and we've got uh the 624 00:30:23,000 --> 00:30:26,120 format of a protocol subdomain uh domain 625 00:30:26,120 --> 00:30:29,519 tldd and our scenario here is that for a 626 00:30:29,519 --> 00:30:31,559 Simplicity of reporting uh we just want 627 00:30:31,559 --> 00:30:33,760 to work with this referred domain field 628 00:30:33,760 --> 00:30:38,320 and be able to simplify that so in 629 00:30:38,320 --> 00:30:41,799 actually splitting out the field in this 630 00:30:41,799 --> 00:30:44,880 case uh split refer domain and then 631 00:30:44,880 --> 00:30:47,720 choosing the period character as our 632 00:30:47,720 --> 00:30:50,399 point to split the data we're creating a 633 00:30:50,399 --> 00:30:52,919 multivalue uh from what was previously 634 00:30:52,919 --> 00:30:57,200 just a a single value field uh and using 635 00:30:57,200 --> 00:31:01,600 this we can actually create a new field 636 00:31:01,600 --> 00:31:06,080 by using the index of a multivalue field 637 00:31:06,080 --> 00:31:08,039 and in this case uh we're looking at 638 00:31:08,039 --> 00:31:09,760 index 639 00:31:09,760 --> 00:31:13,279 012 the multivalue index function allows 640 00:31:13,279 --> 00:31:15,799 us to Target a specific field and then 641 00:31:15,799 --> 00:31:18,559 choose a starting and ending index to 642 00:31:18,559 --> 00:31:21,320 extract given values there are a number 643 00:31:21,320 --> 00:31:23,320 of ways to do this in our case here 644 00:31:23,320 --> 00:31:25,039 where we have three entries it's quite 645 00:31:25,039 --> 00:31:26,639 simple just to give that start and end 646 00:31:26,639 --> 00:31:28,639 of range as the 647 00:31:28,639 --> 00:31:30,039 two entries 648 00:31:30,039 --> 00:31:35,360 apart so as we are working to recreate 649 00:31:35,360 --> 00:31:39,200 our domain and so that is just applying 650 00:31:39,200 --> 00:31:41,720 uh for this new domain field we have 651 00:31:41,720 --> 00:31:44,200 Buttercup games.com and what was 652 00:31:44,200 --> 00:31:48,200 previously the HTTP www. Buttercup 653 00:31:48,200 --> 00:31:51,440 games.com uh we can now use those fields 654 00:31:51,440 --> 00:31:54,720 in a transformation function in this 655 00:31:54,720 --> 00:31:58,039 case simple stats count by status uh in 656 00:31:58,039 --> 00:32:00,200 the 657 00:32:02,600 --> 00:32:06,960 domain so I do want to look at another 658 00:32:06,960 --> 00:32:10,240 uh example here that is similar but 659 00:32:10,240 --> 00:32:13,639 we're going to use a multivalue function 660 00:32:13,639 --> 00:32:16,919 to actually test a condition and so I'm 661 00:32:16,919 --> 00:32:18,399 going 662 00:32:18,399 --> 00:32:21,639 to in this case uh be searching the same 663 00:32:21,639 --> 00:32:24,240 data we're going to start with a stats 664 00:32:24,240 --> 00:32:28,639 command and so a stats count as well as 665 00:32:28,639 --> 00:32:32,039 a values of status and so the values 666 00:32:32,039 --> 00:32:33,360 function is going to provide all the 667 00:32:33,360 --> 00:32:37,480 unique values of a given field uh based 668 00:32:37,480 --> 00:32:41,840 on uh the split by and so that produces 669 00:32:41,840 --> 00:32:44,960 a multivalue field here in the case of 670 00:32:44,960 --> 00:32:47,279 status we have quite a few events uh 671 00:32:47,279 --> 00:32:50,799 that have multiple status codes and as 672 00:32:50,799 --> 00:32:52,960 we're interested in pulling those events 673 00:32:52,960 --> 00:32:57,480 out we can use an MV count function to 674 00:32:57,480 --> 00:33:01,200 eval valate and filter our data set to 675 00:33:01,200 --> 00:33:04,240 those specific events so a very simple 676 00:33:04,240 --> 00:33:07,200 operation here just looking at what has 677 00:33:07,200 --> 00:33:10,240 the uh what has more than a single value 678 00:33:10,240 --> 00:33:13,399 for status uh but very useful as you're 679 00:33:13,399 --> 00:33:15,919 applying this in reporting especially in 680 00:33:15,919 --> 00:33:19,000 combination with others and uh with more 681 00:33:19,000 --> 00:33:22,639 complex conditions 682 00:33:22,639 --> 00:33:28,200 um so uh that is our set of multivalue 683 00:33:28,200 --> 00:33:32,519 eval functions there as streaming 684 00:33:34,240 --> 00:33:38,279 commands so for a uh final section of 685 00:33:38,279 --> 00:33:42,000 the demo I want to talk about a concept 686 00:33:42,000 --> 00:33:44,720 that is not so much a set of functions 687 00:33:44,720 --> 00:33:47,960 uh but really enables uh more complex 688 00:33:47,960 --> 00:33:50,159 and interesting searching and can allow 689 00:33:50,159 --> 00:33:52,799 us to use a few different types of 690 00:33:52,799 --> 00:33:57,240 commands uh in our SPL and so concept of 691 00:33:57,240 --> 00:34:00,200 sub searching for both filtering and 692 00:34:00,200 --> 00:34:04,279 enrichment uh is taking secondary search 693 00:34:04,279 --> 00:34:06,960 results uh and we're using that to 694 00:34:06,960 --> 00:34:10,359 affect a primary search uh so a sub 695 00:34:10,359 --> 00:34:12,200 search will be executed the results 696 00:34:12,200 --> 00:34:15,079 returned and depending on how it's used 697 00:34:15,079 --> 00:34:17,760 uh this is going to be processed in the 698 00:34:17,760 --> 00:34:21,599 original search uh and that is going to 699 00:34:21,599 --> 00:34:24,359 will look at an example that it is 700 00:34:24,359 --> 00:34:27,399 filtering So based on the results we get 701 00:34:27,399 --> 00:34:31,240 a effectively a value equals X or value 702 00:34:31,240 --> 00:34:34,320 equals y uh for one of our fields that 703 00:34:34,320 --> 00:34:37,159 we're looking at in the sub search uh 704 00:34:37,159 --> 00:34:39,320 and then we're also going to look at an 705 00:34:39,320 --> 00:34:42,399 enrichment example so you see this often 706 00:34:42,399 --> 00:34:45,760 when you have uh a data set maybe saved 707 00:34:45,760 --> 00:34:48,480 in a lookup table uh or you just have a 708 00:34:48,480 --> 00:34:50,079 simple reference where you want to bring 709 00:34:50,079 --> 00:34:52,879 in more context maybe descriptions of 710 00:34:52,879 --> 00:34:54,560 event codes things like 711 00:34:54,560 --> 00:34:59,640 that so in that case 712 00:35:02,160 --> 00:35:05,440 we'll look at the First Command here now 713 00:35:05,440 --> 00:35:08,160 I'm going to run my search and we're 714 00:35:08,160 --> 00:35:12,119 going to Pivot over uh to a sub search 715 00:35:12,119 --> 00:35:14,480 tab here and so you can see our sub 716 00:35:14,480 --> 00:35:19,720 search looking at the secure uh logs uh 717 00:35:19,720 --> 00:35:21,880 we are actually just pulling out the 718 00:35:21,880 --> 00:35:24,359 search to see what the results are uh or 719 00:35:24,359 --> 00:35:26,079 what's going to be returned from that 720 00:35:26,079 --> 00:35:28,839 sub search so we're applying the same 721 00:35:28,839 --> 00:35:31,200 rex that we had before to extract our 722 00:35:31,200 --> 00:35:33,720 Fields we're applying a wear a streaming 723 00:35:33,720 --> 00:35:35,920 command looking for anything that's not 724 00:35:35,920 --> 00:35:38,599 null for user we observed that we had 725 00:35:38,599 --> 00:35:40,920 about 60% of our events that were going 726 00:35:40,920 --> 00:35:43,359 to be null based on not having a user 727 00:35:43,359 --> 00:35:46,800 field and so looking at that total data 728 00:35:46,800 --> 00:35:50,280 set uh we're just going to count by our 729 00:35:50,280 --> 00:35:53,839 source IP and this is often a quick way 730 00:35:53,839 --> 00:35:56,839 to really just get a list of unique 731 00:35:56,839 --> 00:35:59,880 values of any given field uh and then 732 00:35:59,880 --> 00:36:03,119 operating on that uh to return just the 733 00:36:03,119 --> 00:36:05,079 the list of values few different ways to 734 00:36:05,079 --> 00:36:08,800 do that uh see stats count pretty often 735 00:36:08,800 --> 00:36:10,599 and in this case we're actually tbling 736 00:36:10,599 --> 00:36:13,960 out just keeping our source IP field and 737 00:36:13,960 --> 00:36:16,800 renaming to client IP so the resulting 738 00:36:16,800 --> 00:36:20,560 data set is a single column table uh 739 00:36:20,560 --> 00:36:21,440 with 740 00:36:21,440 --> 00:36:26,319 182 results and the field name is client 741 00:36:26,319 --> 00:36:29,880 IP so so when returned to the original 742 00:36:29,880 --> 00:36:32,119 search we're running this as a sub 743 00:36:32,119 --> 00:36:36,319 search the effective result of this is 744 00:36:36,319 --> 00:36:39,960 actually client IP equals my first value 745 00:36:39,960 --> 00:36:43,800 here or client IP equals my second value 746 00:36:43,800 --> 00:36:46,960 and so on through the full data set and 747 00:36:46,960 --> 00:36:49,200 so looking at our search here we're 748 00:36:49,200 --> 00:36:52,359 applying this to the access logs you can 749 00:36:52,359 --> 00:36:55,280 see that we had a field named Source IP 750 00:36:55,280 --> 00:36:58,520 in the secure logs uh and we renamed a 751 00:36:58,520 --> 00:37:02,160 client IP so that we could apply this to 752 00:37:02,160 --> 00:37:05,760 the access logs where client IP is the 753 00:37:05,760 --> 00:37:09,480 actual field name for the uh Source IP 754 00:37:09,480 --> 00:37:13,560 data and in this case we are filtering 755 00:37:13,560 --> 00:37:16,079 to the client IPS relevant in the secure 756 00:37:16,079 --> 00:37:19,839 logs for our web access 757 00:37:19,839 --> 00:37:23,960 logs so uncommenting here we have a 758 00:37:23,960 --> 00:37:26,800 series of operations that we're doing uh 759 00:37:26,800 --> 00:37:29,000 and I'm just going to run the mall at 760 00:37:29,000 --> 00:37:33,079 once and talk through uh that we are 761 00:37:33,079 --> 00:37:37,240 counting uh the status or we're counting 762 00:37:37,240 --> 00:37:40,319 the events by status and client IP uh 763 00:37:40,319 --> 00:37:42,640 for the client IPS that were relevant to 764 00:37:42,640 --> 00:37:44,880 authentication failures in the secure 765 00:37:44,880 --> 00:37:48,760 logs we are then creating a status count 766 00:37:48,760 --> 00:37:52,040 field just by combining uh our status 767 00:37:52,040 --> 00:37:54,680 and count Fields uh adding a colant 768 00:37:54,680 --> 00:37:58,640 between them uh and then we are doing a 769 00:37:58,640 --> 00:38:02,079 second uh stats statement here to 770 00:38:02,079 --> 00:38:03,960 actually combine all of our newly 771 00:38:03,960 --> 00:38:06,319 created Fields together in a more 772 00:38:06,319 --> 00:38:10,560 condensed report so transforming command 773 00:38:10,560 --> 00:38:12,520 then streaming for creating our new 774 00:38:12,520 --> 00:38:15,359 field another transforming command and 775 00:38:15,359 --> 00:38:17,880 then our sort for data set processing 776 00:38:17,880 --> 00:38:20,920 actually gives us the results here for a 777 00:38:20,920 --> 00:38:25,480 given client IP and so we are in this 778 00:38:25,480 --> 00:38:28,440 case looking for the scenario that 779 00:38:28,440 --> 00:38:31,319 these client IPS that are involved in 780 00:38:31,319 --> 00:38:34,240 authentication failures to the web 781 00:38:34,240 --> 00:38:37,319 servers in this case these were all over 782 00:38:37,319 --> 00:38:39,680 SSH uh we want to see if there are 783 00:38:39,680 --> 00:38:42,760 interactions by these same Source IPS uh 784 00:38:42,760 --> 00:38:46,079 actually on the uh website that we're 785 00:38:46,079 --> 00:38:50,200 hosting uh so seeing a high number of 786 00:38:50,200 --> 00:38:53,400 failed values looking at actions also is 787 00:38:53,400 --> 00:38:55,599 a use case here for just bringing in 788 00:38:55,599 --> 00:38:57,680 that context and seeing if there's any 789 00:38:57,680 --> 00:39:00,520 sort of relationship between the data uh 790 00:39:00,520 --> 00:39:04,000 this is discussed often as correlation 791 00:39:04,000 --> 00:39:07,680 of logs I'm usually careful about using 792 00:39:07,680 --> 00:39:09,440 the term correlation in talking about 793 00:39:09,440 --> 00:39:11,119 spun queries especially in Enterprise 794 00:39:11,119 --> 00:39:12,640 security talking about correlation 795 00:39:12,640 --> 00:39:16,119 searches where I typically think of 796 00:39:16,119 --> 00:39:18,480 correlation searches as being 797 00:39:18,480 --> 00:39:20,599 overarching Concepts that cover data 798 00:39:20,599 --> 00:39:23,920 from multiple data sources and in this 799 00:39:23,920 --> 00:39:26,480 case correlating events would be looking 800 00:39:26,480 --> 00:39:28,400 at unique data types that are 801 00:39:28,400 --> 00:39:31,240 potentially related uh in finding that 802 00:39:31,240 --> 00:39:33,839 logical connection uh for the condition 803 00:39:33,839 --> 00:39:35,880 that's a little bit more up to the user 804 00:39:35,880 --> 00:39:38,319 it's not uh quite as easy as say 805 00:39:38,319 --> 00:39:41,520 pointing to a specific data 806 00:39:41,520 --> 00:39:44,880 model so we are going to look at one 807 00:39:44,880 --> 00:39:47,920 more sub search here and this case is 808 00:39:47,920 --> 00:39:52,240 going to apply uh the join command and 809 00:39:52,240 --> 00:39:55,680 so I talk about using lookup files uh or 810 00:39:55,680 --> 00:39:59,000 uh other data returned by sub searches 811 00:39:59,000 --> 00:40:01,599 uh to enrich to bring more data in 812 00:40:01,599 --> 00:40:05,599 rather than filter um we are going to 813 00:40:05,599 --> 00:40:08,960 look at our first part of the command 814 00:40:08,960 --> 00:40:11,480 here uh and this is actually just a 815 00:40:11,480 --> 00:40:15,720 simple uh stats report based on this rex 816 00:40:15,720 --> 00:40:18,079 that keeps coming through uh the SPL to 817 00:40:18,079 --> 00:40:21,000 give us those user and Source IP Fields 818 00:40:21,000 --> 00:40:24,079 uh so our result here is authentication 819 00:40:24,079 --> 00:40:26,200 failures for all these web hosts so 820 00:40:26,200 --> 00:40:28,760 similar to what we had previously 821 00:40:28,760 --> 00:40:31,200 returned and then we're going to take a 822 00:40:31,200 --> 00:40:33,319 look at the results of the sub search 823 00:40:33,319 --> 00:40:35,400 here actually split this up so that we 824 00:40:35,400 --> 00:40:38,839 can see uh the first two lines we're 825 00:40:38,839 --> 00:40:41,760 looking at our web access logs for 826 00:40:41,760 --> 00:40:45,560 purchase actions uh and then we are 827 00:40:45,560 --> 00:40:50,599 looking at uh our stats count for errors 828 00:40:50,599 --> 00:40:52,960 and stats count for successes we have 829 00:40:52,960 --> 00:40:55,079 pretty limited status code to return in 830 00:40:55,079 --> 00:40:59,240 this data so this uh is is uh viable for 831 00:40:59,240 --> 00:41:01,800 the data present uh to observe our 832 00:41:01,800 --> 00:41:03,280 errors and 833 00:41:03,280 --> 00:41:05,880 successes and then we are actually 834 00:41:05,880 --> 00:41:08,160 creating a new field based on the 835 00:41:08,160 --> 00:41:10,839 statistics that we're generating uh 836 00:41:10,839 --> 00:41:13,920 looking at our transaction errors so 837 00:41:13,920 --> 00:41:18,000 where we have uh high or low numbers uh 838 00:41:18,000 --> 00:41:22,079 of failed purchase actions uh and then 839 00:41:22,079 --> 00:41:25,599 summarizing that so in the case of our 840 00:41:25,599 --> 00:41:27,800 final command here another transforming 841 00:41:27,800 --> 00:41:30,640 command of table just to reduce this to 842 00:41:30,640 --> 00:41:35,079 a small data set uh to use in the subur 843 00:41:35,079 --> 00:41:37,440 and so in this case we have our host 844 00:41:37,440 --> 00:41:39,400 value and then our transaction error 845 00:41:39,400 --> 00:41:41,480 rate that we observe from the web access 846 00:41:41,480 --> 00:41:44,760 logs and then over in our other search 847 00:41:44,760 --> 00:41:48,640 here uh we are going to perform a left 848 00:41:48,640 --> 00:41:51,400 join based on this host field so you see 849 00:41:51,400 --> 00:41:53,359 in our secure logs we still have the 850 00:41:53,359 --> 00:41:55,800 same host value and this is going to be 851 00:41:55,800 --> 00:41:59,640 used uh to to actually add our 852 00:41:59,640 --> 00:42:02,760 transaction uh error rates in for each 853 00:42:02,760 --> 00:42:06,400 host so as we observe uh increased 854 00:42:06,400 --> 00:42:08,640 authentication failures if there's a 855 00:42:08,640 --> 00:42:11,960 scenario for a breach and some sort of 856 00:42:11,960 --> 00:42:14,960 interruption to the ability to serve out 857 00:42:14,960 --> 00:42:17,520 or perform these purchase actions that 858 00:42:17,520 --> 00:42:20,960 that are affecting uh the intended 859 00:42:20,960 --> 00:42:23,200 operations of the web servers uh we can 860 00:42:23,200 --> 00:42:25,280 see that here of course our tutorial 861 00:42:25,280 --> 00:42:27,319 data there's not really much that 862 00:42:27,319 --> 00:42:29,880 jumping out or showing uh that there is 863 00:42:29,880 --> 00:42:32,599 any correlation between the two but the 864 00:42:32,599 --> 00:42:34,640 purpose of the join is to bring in that 865 00:42:34,640 --> 00:42:37,440 extra data set to give the context to 866 00:42:37,440 --> 00:42:39,839 further 867 00:42:41,040 --> 00:42:47,440 investigate so um that is uh the final 868 00:42:47,440 --> 00:42:52,359 portion of the SPL demo uh and I do want 869 00:42:52,359 --> 00:42:54,920 to say for any questions I'm going to 870 00:42:54,920 --> 00:42:56,960 take a look at the chat I'll do my best 871 00:42:56,960 --> 00:43:00,079 to answer any questions um and then if 872 00:43:00,079 --> 00:43:03,079 you have any other questions uh please 873 00:43:03,079 --> 00:43:05,800 feel free to reach out to my team at 874 00:43:05,800 --> 00:43:08,599 support keny group.com and we'll be 875 00:43:08,599 --> 00:43:11,920 happy to get back to you and help um I 876 00:43:11,920 --> 00:43:15,440 am taking a look 877 00:43:26,119 --> 00:43:29,119 through 878 00:43:32,200 --> 00:43:33,760 okay seeing some questions on 879 00:43:33,760 --> 00:43:38,280 performance of the uh Rex said Rex 880 00:43:38,280 --> 00:43:41,599 commands um so off the top of my head I 881 00:43:41,599 --> 00:43:43,800 I'm not sure about a direct performance 882 00:43:43,800 --> 00:43:46,400 comparison uh of the individual commands 883 00:43:46,400 --> 00:43:49,200 definitely want to look into that um and 884 00:43:49,200 --> 00:43:52,280 definitely follow up uh if you'd like to 885 00:43:52,280 --> 00:43:54,280 uh explain a more detailed scenario or 886 00:43:54,280 --> 00:43:57,119 look at some SPL uh that we can apply in 887 00:43:57,119 --> 00:43:58,400 observe those 888 00:43:58,400 --> 00:44:01,680 changes um the question on getting the 889 00:44:01,680 --> 00:44:05,480 data set uh that is what I mentioned at 890 00:44:05,480 --> 00:44:07,520 the beginning uh reach out to us for the 891 00:44:07,520 --> 00:44:10,119 slides uh or just uh reach out about the 892 00:44:10,119 --> 00:44:15,480 link and the uh Splunk tutorial data you 893 00:44:15,480 --> 00:44:17,880 can actually search that as well um and 894 00:44:17,880 --> 00:44:20,400 there's documentation on how to use the 895 00:44:20,400 --> 00:44:22,400 tutorial data one of the first links 896 00:44:22,400 --> 00:44:25,640 there uh takes you to a page that has uh 897 00:44:25,640 --> 00:44:29,079 it is a tutorial data zip file uh and 898 00:44:29,079 --> 00:44:31,079 instructions on how to injust that it's 899 00:44:31,079 --> 00:44:34,079 just an upload uh for your specific 900 00:44:34,079 --> 00:44:37,599 environment so uh in add data and then 901 00:44:37,599 --> 00:44:40,040 upload data two clicks uh and upload 902 00:44:40,040 --> 00:44:43,400 your file so that is uh freely available 903 00:44:43,400 --> 00:44:45,760 for anyone uh and again that package is 904 00:44:45,760 --> 00:44:47,440 dynamically updated as well so your time 905 00:44:47,440 --> 00:44:51,359 stamps are pretty close to to normal uh 906 00:44:51,359 --> 00:44:53,440 as you download the app kind of depends 907 00:44:53,440 --> 00:44:55,920 on the time of the the cycle for the 908 00:44:55,920 --> 00:44:58,559 update um but search overall time you 909 00:44:58,559 --> 00:45:02,359 won't have any issues there um and then 910 00:45:02,359 --> 00:45:05,119 yeah again on receiving slides uh reach 911 00:45:05,119 --> 00:45:08,240 out to my team uh and we're happy to to 912 00:45:08,240 --> 00:45:10,240 provide those discuss further and we'll 913 00:45:10,240 --> 00:45:16,040 have uh the um the recording available 914 00:45:16,040 --> 00:45:18,400 for this session you should be able to 915 00:45:18,400 --> 00:45:20,680 after uh the recording processes when 916 00:45:20,680 --> 00:45:22,880 the session ends uh actually use the 917 00:45:22,880 --> 00:45:24,640 same link and you can watch this 918 00:45:24,640 --> 00:45:26,480 reporting and post uh without having to 919 00:45:26,480 --> 00:45:31,800 sign up or transfer that file so 920 00:45:33,680 --> 00:45:38,319 um so okay Chris seeing uh seeing your 921 00:45:38,319 --> 00:45:41,240 comment there um let me know if you want 922 00:45:41,240 --> 00:45:44,480 to reach out to me directly anyone as 923 00:45:44,480 --> 00:45:49,440 well um we can discuss what slides and 924 00:45:49,440 --> 00:45:51,640 presentation you had attended I'm not 925 00:45:51,640 --> 00:45:55,359 sure I have the attendance report uh for 926 00:45:55,359 --> 00:45:57,319 for what You' seen previously so uh 927 00:45:57,319 --> 00:46:00,240 happy to get those for 928 00:46:06,720 --> 00:46:10,319 you all right and uh seeing thanks Brett 929 00:46:10,319 --> 00:46:13,079 so you see Brett Woodruff in the chat 930 00:46:13,079 --> 00:46:16,680 commenting uh systems engineer on the uh 931 00:46:16,680 --> 00:46:18,640 expertise on demand team so very 932 00:46:18,640 --> 00:46:20,400 knowledgeable guy and he's going to be 933 00:46:20,400 --> 00:46:23,720 presenting next month's session uh that 934 00:46:23,720 --> 00:46:25,400 is going to take this concept that we 935 00:46:25,400 --> 00:46:28,760 talked about the subur in as a just 936 00:46:28,760 --> 00:46:30,760 general search topic he's going to go 937 00:46:30,760 --> 00:46:34,319 specifically into Data enrichment using 938 00:46:34,319 --> 00:46:38,079 uh joins lookup commands and how we see 939 00:46:38,079 --> 00:46:41,079 uh that used in the wild so definitely 940 00:46:41,079 --> 00:46:43,359 excited for that one encourage you to 941 00:46:43,359 --> 00:46:46,480 register for for that 942 00:46:46,920 --> 00:46:52,240 event all right I'm not seeing any more 943 00:46:55,839 --> 00:46:57,800 questions 944 00:46:57,800 --> 00:47:02,119 all right with that uh I am stopping my 945 00:47:02,119 --> 00:47:05,079 share I'm going to hang around for a few 946 00:47:05,079 --> 00:47:07,440 minutes uh but thank you all for 947 00:47:07,440 --> 00:47:11,079 attending and we'll see you on the next 948 00:47:15,920 --> 00:47:18,920 session