Campaign Manager Phase 2(Scrubbing) - Technical Documentation

Requirement

Our customers follow specific rules daily before uploading data to our central platform. This repetitive process consumes significant time and leads to inefficiencies.

As a user, I want the ability to create and manage fully customizable data scrubbing filters, so I can define specific validation rules and control system functionality to meet my unique data validation requirements. To address this, a new section, "Scrubbing Management," should be introduced, allowing users to configure and manage scrubbing filters that meet their unique data validation requirements.

Ex . The customer will create a filter based on mobile numbers and upload a file containing 100,000 mobile numbers for filtering. After the filter is created, the customer will upload a file with 500,000 records and apply the filter to the uploaded file. During the processing of the uploaded campaign file, our system will mark 100,000 records as 'SCRUBBED' and will not process them, while the remaining 400,000 records will be processed.

Technical Approach

The file processing on file data will process for KFP and store in storage(Redis).
The data filtering process will be applied on KFP.
All filter TLL must be set in storage, after expiry will be removed from storage.
We need to add limitations on active filters.
We need to add a limit on the apply filter at the time of uploading the file.
At the time of adding condition users can choose Regular Expression OR Operator.
Neet to set limits for adding conditions.
Users can upload scrubbing data files multiple times if active.
Neet to remove all filter data when the user deletes the filter.
Show all filter lists on Scrubbing Manager if TTL expired ??

Module effected :

Web UI
Kairos-Service.
Support-UI
KFP
KPM
Kairos-report

New Table Required :

ScrubManagerFilter
FileterRule
ScrubFileUpload

Modify Table

UploadedFileLog

Detail Description :

ScrubManagerFilter

Column Name	Data Type	Null/Not Null
filterId	int	Primary key
filterName	varchar(100)	Not Null
validity	dateTime	Not Null
status	int	Not Null
accountId	Int	Not Null
createTime	dateTime	Not Null

FileterRule

Column Name	Data Type	Null/Not Null
ruleId	int	Primary key
filterId	int	Foreign Key
keyVariableName	varchar(100)	Null
ConditionType	enum	OR/AND
regularExpression	Text	Null
operator	varchar(50)	Null

ScrubFileUpload

Column Name	Data Type	Null/Not Null
fileId	int	Primary key
actualFileName	varchar(255)	Not Null
fileName	varchar(255)	Null
size	int	Null
header	Text	Not Null
uploadedRecords	int	Null
fileCountRecords	int	Null
statusId	int	Not Null
processStartTime	dateTime	Null
processEndTime	dateTime	Null
canceledTime	dateTime	Null
ipAddress	varchar(50)	Not Null
uploadedBy	varchar(255)	Not Null

KFP File Processing

KPM Process

Phonon Software

Campaign Manager Phase 2(Scrubbing) - Technical Documentation

Analytics

Requirement

Related content