Examples do not work in distributed search

[Bre77]

The example code for Streaming Search commands, does not work in a distributed search environment, because the %app%/lib directory is not included in bundle replication.

splunk-sdk-python/examples/searchcommands_template/bin/stream.py

Line 6 in 2199b5e

sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "lib"))

The lib directory needs to instead be located inside bin, or a distsearch.conf should be included, or I guess a note about it has to be installed on the indexers just so it has access to Splunklib.

[vmalaviya-splunk]

Hey @Bre77
We'll need a couple of details to reproduce the issue.

1. Splunk version
2. Steps to create a replica of your distributed environment.

[Bre77]

Hey @Bre77
We'll need a couple of details to reproduce the issue.

1. Splunk version
2. Steps to create a replica of your distributed environment.

I have tested this in both the current Splunk Cloud version and the latest Splunk Enterprise version. My distributed environment was an index cluster with the TA only installed on the search head.

Issue can easily be reproduced by installing any version 0.1.x of https://splunkbase.splunk.com/app/6161/ on a search head with distributed search. I fixed this issue in 1.0.0 by moving the lib directory inside bin.

[Bre77]

I believe this documentation needs to be implemented into the example: https://dev.splunk.com/enterprise/docs/devtools/customsearchcommands/packageanddeploycustom/#Deploy-a-streaming-command-on-an-indexer

[tacole02]

Thanks so much for raising this issue, @Bre77

The correct solution is so include a distsearch.conf file with the /lib directory added to the replication allow list.

[bparmar-splunk]

Hi @Bre77,
We have tried to reproduce this issue (using docker) with architecture mentioned below:

• Splunk latest
• 3 search heads
• 3 indexers

We have installed (.tar of) searchcommands_app (comprises of several custom search commands) located in python-sdk repo.
All custom commands are tested and working fine. We could not encounter any issue while testing.
App directory structure used here as:

searchcommands_app/

bin/
data/
default/
lib/ (Contains splunklib folder)
lookups/
metadata/
README/
searchcommands_app.log

Please let us know, if we missed anything.

[Bre77]

@bparmar-splunk where was the search command installed in your test? Only Search Heads or on the indexers too?

[bparmar-splunk]

@Bre77,
We have tested using virtual infrastructure of above mentioned configuration and search command was installed on all nodes.
We have verified on all nodes using Splunk Web UI and it was working fine.

[Bre77]

@bparmar-splunk thats why you couldn't reproduce. Try installing a streaming search command only on the search head. It will be distributed to the indexers in the knowledge bundle, but be missing the splunklib directory because the provided example does not include a distsearch.conf as per the Splunk Dev docs: https://dev.splunk.com/enterprise/docs/devtools/customsearchcommands/packageanddeploycustom/#Deploy-a-streaming-command-on-an-indexer