Splunk interview questions, with answers that show understanding.

Data is read on the source by a forwarder, which sends it to an indexer. The indexer parses it, assigns a timestamp, breaks it into events, and writes it to an index. When a user searches, the search head queries the indexers and assembles the result. Naming those four parts — forwarder, indexer, search head, index — and what each does is the core of the answer.

A universal forwarder is lightweight and mostly just ships raw data; it is what you put on most endpoints. A heavy forwarder can parse and route data before sending it on, at the cost of more resources. Choose the universal forwarder unless you specifically need pre-processing close to the source.

It means Splunk does not force data into a fixed structure when it is stored; fields are extracted at search time. The benefit is flexibility — you can ingest messy data quickly and decide how to interpret it later, which suits investigations where you do not know in advance what you will need.

Start narrow, then summarise. Something like:

index=auth action=failure
| stats count by user
| sort - count

The first line filters to the relevant events, stats aggregates them by user, and sort puts the noisiest accounts on top. Being able to explain each pipe is what the interviewer is checking.

stats collapses events into a summary table. eventstats computes the same aggregate but adds it back onto every original event as a new field, so you keep the raw events. Use eventstats when you want to compare each event against a group total.

Filter as early as possible: specify the index, a tight time range, and known field values up front so less data is scanned. Avoid wildcards at the start of terms, and push transforming commands like stats to the end. For repeated reporting, consider a summary or accelerated data model.

A sourcetype labels the format of incoming data so Splunk knows how to parse timestamps and fields. Getting it right at onboarding is important because a wrong sourcetype leads to broken timestamps and missing fields, which quietly undermines every search and detection built on top.

Check the sourcetype's timestamp configuration — the time format and which field the time is read from. Misparsed timestamps usually mean events are landing under the wrong time, so start there before suspecting the data itself.

Confirm the forwarder service is running, check it can reach the indexer over the network and port, and look at the forwarder's own logs for errors. Then verify on the indexer side that the input is configured and the host is not being filtered. Work the path end to end rather than guessing.

Find which panel is slow and inspect its search. Often the fix is the same as for any slow search: tighter time range, better filtering, or backing the panel with a summary instead of a raw search over a huge window. Measure before and after so you can show the improvement.