Data Cleaning Utility

This section contains a text replacement implementation that is fast and scalable. More...

Typedefs
typedef void *	ruCleaner
typedef rusize_s(*	rcWriteFn) (perm_ptr ctx, trans_ptr buf, rusize len)
typedef rusize_s(*	rcReadFn) (perm_ptr ctx, ptr buf, rusize len)
typedef void(*	ruCleanerCb) (perm_ptr user_data, trans_chars key, trans_chars subst)

Functions
RUAPI ruCleaner	ruCleanNew (rusize chunkSize)
	Creates a new ruCleaner object. To be freed with ruCleanFree. More...
RUAPI ruCleaner	ruCleanFree (ruCleaner rc)
RUAPI int32_t	ruCleanDump (ruCleaner cp, ruCleanerCb lf, perm_ptr user_data)
RUAPI int32_t	ruCleanAdd (ruCleaner rc, trans_chars instr, trans_chars substitute)
RUAPI int32_t	ruCleanRemove (ruCleaner rc, trans_chars instr)
RUAPI int32_t	ruCleanIo (ruCleaner rc, rcReadFn reader, perm_ptr readCtx, rcWriteFn writer, perm_ptr writeCtx)
	Does replacements using the given I/O functions. More...
RUAPI int32_t	ruCleanToWriter (ruCleaner rc, trans_chars in, rusize len, rcWriteFn writer, perm_ptr writeCtx)
	Does replacements on given string using the given Output function. More...
RUAPI int32_t	ruCleanToString (ruCleaner rc, trans_chars in, rusize len, ruString *out)
	Does replacements in given string and returns out in given String Object. More...

Detailed Description

This section contains a text replacement implementation that is fast and scalable.

It is meant to be used to clean credentials out of large files in a minimum amount of time.

Example:

#include <stdio.h>
#include <regify-util.h>
 
rusize_s reader(const void* in, void* data, rusize len) {
    return (rusize_s) fread(data, len, 1, (FILE*)in);
}
 
rusize_s writer(const void* out, const void* data, rusize len) {
    return (rusize_s) fwrite(data, len, 1, (FILE*)out);
}
 
void clean() {
    int32_t ret;
    FILE *in = 0, *out = 0;
    ruCleaner rc = ruCleanNew(0);
    const char* repl = "^^^PASSWORD^^^";
    do {
        ret = ruCleanAdd(rc, "mysecret", repl);
        if (ret != RUE_OK) {
            printf("Failed adding secret: %d", ret);
            break;
        }
        ret = ruCleanAdd(rc, "myothersecret", repl);
        if (ret != RUE_OK) {
            printf("Failed adding other secret: %d", ret);
            break;
        }
        in = ruFOpen("debug.log", "r", &ret);
        if (ret != RUE_OK) {
            printf("Failed opening debug.log: %d", ret);
            break;
        }
        out = ruFOpen("clean.log", "w", &ret);
        if (ret != RUE_OK) {
            printf("Failed opening clean.log: %d", ret);
            break;
        }
        ret = ruCleanIo(rc, &reader, in, &writer, out);
        if (ret != RUE_OK) {
            printf("Failed setting I/O context: %d", ret);
            break;
        }
        ret = ruCleanNow(rc);
        if (ret != RUE_OK) {
            printf("Failed cleaning log: %d", ret);
            break;
        }
    } while(0);
 
    if (in) fclose(in);
    if (out) fclose(out);
    ruCleanFree(rc);
}

Typedef Documentation

rcReadFn

typedef rusize_s(* rcReadFn) (perm_ptr ctx, ptr buf, rusize len)

The type function for the read operation when cleaning a data stream.

rcWriteFn

typedef rusize_s(* rcWriteFn) (perm_ptr ctx, trans_ptr buf, rusize len)

The type function for the write operation when cleaning a data stream.

ruCleaner

typedef void* ruCleaner

Opaque pointer to cleaner object. See Data Cleaning Utility

ruCleanerCb

typedef void(* ruCleanerCb) (perm_ptr user_data, trans_chars key, trans_chars subst)

A callback for the ruCleanDump function. The given key and subst parameters belong to the cleaner and must not be freed.

Parameters

user_data	The optional context that was given to ruCleanDump.
key	The secret key to clean.
subst	The placeholder to substitute the secret with.

Function Documentation

ruCleanAdd()

RUAPI int32_t ruCleanAdd	(	ruCleaner	rc,
		trans_chars	instr,
		trans_chars	substitute
	)

Adds a new string to be replaced.

Parameters

rc	The relevant ruCleaner object
instr	The string to be replaced. Will be copied.
substitute	The string to replace instr with. Will be copied.

Returns

RUE_OK on success else an error code.

ruCleanDump()

RUAPI int32_t ruCleanDump	(	ruCleaner	cp,
		ruCleanerCb	lf,
		perm_ptr	user_data
	)

Iterates over the cleaner database and calls given ruCleanerCb with each secret and substitute. This function is mainly used for debugging.

Parameters

cp	The relevant ruCleaner object
lf	The ruCleanerCb to call with each entry
user_data	An optional context that will be given to the callbacks user_data parameter.

Returns

RUE_OK on success else an error code.

ruCleanFree()

RUAPI ruCleaner ruCleanFree

(

ruCleaner

rc

)

Frees the given ruCleaner object.

Parameters

rc	list to free.

Returns

NULL

ruCleanIo()

RUAPI int32_t ruCleanIo	(	ruCleaner	rc,
		rcReadFn	reader,
		perm_ptr	readCtx,
		rcWriteFn	writer,
		perm_ptr	writeCtx
	)

Does replacements using the given I/O functions.

This can be an opened file with the corresponding file handle as context. This function may be called multiple times and interchangeably with the ruCleanTo* functions.

Parameters

rc	The relevant ruCleaner object
reader	The read function to be called for data input.
readCtx	The context to be passed to the read function.
writer	The write function to be called for data output.
writeCtx	The context to be passed to the write function.

Returns

RUE_OK on success else an error code.

ruCleanNew()

RUAPI ruCleaner ruCleanNew

(

rusize

chunkSize

)

Creates a new ruCleaner object. To be freed with ruCleanFree.

Unless the ruCleaner comes from a pwcleaner library which lacks the remaining regify-util functionality, it is made thread safe by the internal use of a mutex. pwcleaner users need to do their own locking in multi threaded use cases.

Parameters

chunkSize

Size of chunk to process at a time. Will be allocated twice. Will be increased to the largest item to clean if that is bigger. Defaults to 1M if set to 0.

Returns

Guaranteed to return new ruCleaner object, or process abort.

ruCleanRemove()

RUAPI int32_t ruCleanRemove	(	ruCleaner	rc,
		trans_chars	instr
	)

Removes a string entry from the database.

Parameters

rc	The relevant ruCleaner object
instr	The string to be removed

Returns

RUE_OK on success else an error code.

ruCleanToString()

RUAPI int32_t ruCleanToString	(	ruCleaner	rc,
		trans_chars	in,
		rusize	len,
		ruString *	out
	)

Does replacements in given string and returns out in given String Object.

This function may be called multiple times and interchangeably with the ruCleanTo*, ruCleanIo functions. This function is thread safe but locks allow only one thread at time.

Parameters

rc	The relevant ruCleaner object
in	String to replace tokens in.
len	Length of the string or 0 to use the null terminator for length detection.
out	Where the String Object containing the result will be stored. Caller must free with ruStringFree after use.

Returns

RUE_OK on success else an error code.

ruCleanToWriter()

RUAPI int32_t ruCleanToWriter	(	ruCleaner	rc,
		trans_chars	in,
		rusize	len,
		rcWriteFn	writer,
		perm_ptr	writeCtx
	)

Does replacements on given string using the given Output function.

This can be an opened file with the corresponding file handle as context. This function may be called multiple times and interchangeably with the ruCleanTo*, ruCleanIo functions. This function is thread safe but locks allow only one thread at time.

Parameters

rc	The relevant ruCleaner object
in	String to replace tokens in.
len	Length of the string or 0 to use the null terminator for length detection.
writer	The write function to be called for data output.
writeCtx	The context to be passed to the write function.

Returns

RUE_OK on success else an error code.