Split a string by a delimiter

Tokenizing a string in C typically involves breaking a string into smaller pieces or tokens based on a specified delimiter. The standard library provides a function called strtok that can be used for this purpose. Here's a simple example with comments explaining each step:

#include <stdio.h>
#include <string.h>

int main() {
    // Input string to tokenize
    char inputString[] = "Hello,World,Tokenization,Example";

    // Delimiter used for tokenization (comma in this case)
    const char delimiter[] = ",";

    // Token pointer to store the current token during iteration
    char *token;

    // Use strtok to get the first token
    token = strtok(inputString, delimiter);

    // Continue getting tokens until strtok returns NULL
    while (token != NULL) {
        // Print the current token
        printf("Token: %s\n", token);

        // Use strtok with NULL to continue tokenization
        token = strtok(NULL, delimiter);
    }

    return 0;
}

In the provided C code, a string is tokenized using the strtok function, which breaks the input string, "Hello,World,Tokenization,Example," into smaller pieces based on the specified delimiter (,). The program iterates through the resulting tokens, printing each one on a new line. The tokenization process begins by calling strtok with the input string and delimiter, and subsequent calls use NULL as the first argument to continue tokenization. The token pointer is updated during each iteration, and the loop continues until there are no more tokens. This example demonstrates a basic tokenization scenario, but it's important to note that strtok modifies the original string, and complex cases may require additional handling and error checks.