You may be working around this by allowing development to occur on production or you may have a development search head that peers your prod indexers. However, you notice two impacts of this choice over time.
Even if workload management has been implemented as a good stopgap, isolating testing and production environments from each other is the best practice for change control, security, and performance.
Approach No. 1: Go with a Splunk built tool like Event Gen.
The downsides to this approach?
Approach No. 2: Forward a sample of the data from production to testing.
The big concern here will be a CPU or network performance hit. Only a sample of the data is needed, so forwarding from a single indexer may be enough. Forwarding from 1 indexer on a 12 indexer cluster would create a 1/12th sampling of the data which should create almost no performance impact because forwarded data is not parsed and should only consume a few MBs on a GB+ network
Since the destination indexer is in a testing environment, downtime can be business as usual. The outputs.conf will direct events to be dropped when it gets full. An app containing an outputs.conf similar to the below should be placed on a single prod indexer to forward data or on multiple production indexers for a larger sample. This should be placed in an app within $SPLUNK_HOME$/etc/apps and not pushed from the cluster master. The port number can be changed but should match on sending and receiving indexer. Cooked data is sent to preserve host, source, and sourcetype metadata.
Outputs.conf on the forwarding production indexer would look like:
Inputs.conf on the receiving, non-production indexer would look like:
The queue on the custom input is set to parsingQueue. Normally Splunk will send S2S(Splunk to Splunk) data straight to the index queue, but that would bypass any ingestion rules on the test indexer. In this case, you may want to test ingestion configurations. To do so, the queue is explicitly set to the 'parsingQueue’ so all events pass through the entire pipeline. An alternative port should be used to ensure this behavior is not applied to other S2S forwarded data.
To select which indexes are forwarded, use:
If test indexers are clustered, specify multiple output destinations under server=in the tcpout:name stanza.
Sample data is shown below for both methods.
Note: requires [tcp://<port number>] stanza on receiving side
Decide for yourself which you prefer. Since the sourcetype is retained with cooked data, search head apps and add-ons can use the data as is.
As always, our security experts are here to help. If you're still stuck, drop us an email!
Filtering forwarded data:
Masking sensitive data: