Requirement
Our customers follow specific rules daily before uploading data to our central platform. This repetitive process consumes significant time and leads to inefficiencies.
...
Ex . The customer will create a filter based on mobile numbers and upload a file containing 100,000 mobile numbers for filtering. After the filter is created, the customer will upload a file with 500,000 records and apply the filter to the uploaded file. During the processing of the uploaded campaign file, our system will mark 100,000 records as 'SCRUBBED' and will not process them, while the remaining 400,000 records will be processed.
Technical Approach
The file processing on file data will process for KFP and store in storage(Redis).
The data filtering process will be applied on KFP.
All filter TLL must be set in storage, after expiry will be removed from storage.
We need to add limitations on active filters.
We need to add a limit on the apply filter at the time of uploading the file.
At the time of adding condition users can choose Regular Expression OR Operator.
Neet to set limits for adding conditions.
Users can upload scrubbing data files multiple times if active.
Neet to remove all filter data when the user deletes the filter.
Show all filter lists on Scrubbing Manager if TTL expired ??
...
Detail Description :
ScrubManagerFilter
Column Name | Data Type | Null/Not Null |
filterId | int | Primary key |
filterName | varchar(100) | Not Null |
validity | dateTime | Not Null |
status | int | Not Null |
accountId | Int | Not Null |
createTime | dateTime | Not Null |
FileterRule
Column Name | Data Type | Null/Not Null |
ruleId | int | Primary key |
filterId | int | Foreign Key |
keyVariableName | varchar(100) | Null |
ConditionType | enum | OR/AND |
regularExpression | Text | Null |
operator | varchar(50) | Null |
ScrubFileUpload
Column Name | Data Type | Null/Not Null |
fileId | int | Primary key |
actualFileName | varchar(255) | Not Null |
fileName | varchar(255) | Null |
size | int | Null |
header | Text | Not Null |
uploadedRecords | int | Null |
fileCountRecords | int | Null |
statusId | int | Not Null |
processStartTime | dateTime | Null |
processEndTime | dateTime | Null |
canceledTime | dateTime | Null |
ipAddress | varchar(50) | Not Null |
uploadedBy | varchar(255) | Not Null |
KFP File Processing
KPM Process
...