1. Approach
This approach is based on the management of FDD Document based on the campaign upload requirement.
Flow Details Document CREATION: (Campaign Upload)
When a campaign is uploaded we will create one FDD for each slot with “scheduled_start_time” for each individual record uploaded in the campaign for the present day.
For example, if we have 10,000 records in a file and 3 slots, we will create 10,000 Flow Details Documents for each slot, resulting in a total of 30,000 Flow Details Documents for a single file for the day of the upload.
These Flow Details Documents will be queued for execution according to the slot start time. Additionally, we will add a new variable to the Flow Details Documents to denote the slot.
For each subsequent day the campaign is scheduled to run, we will create Flow Details Documents for each record in a similar manner at 00 hours. ( Flow Details Documents will be created for a single day at a time to prevent overwhelming the database or complicating the overall process)
Note
If no slot is selected at the time of uploading the file then FDD will create without slot and work the same as current flow.
Time Slot Functionality will be working only for those campaigns uploaded from our system.
We are provisionally proposing limiting the maximum number of time slots in one day to 2 and the campaign will be scheduled only for at the most 30 days.
Slot time must be under working time (Flow working time).
If the csv file contains the schedule time field and campaign created in slot then schedule time will be slot start time that means ignore schedule time fields.If the number of records processed within a given slot are more than the slot time then the records which are not processed within slot time will be considered as failed.
All file upload process and scheduler for slot will be processed on KFP.
Upload Negative Number base:
Negative numbers uploaded will be stored in Mysql corresponding to request_id
When the negative numbers are uploaded the corresponding entries will be updated in the respective FDDs and will also check if the numbers are already queued in KP awaiting execution.
When the scheduler runs to create FDD for subsequent days (at 00:00 hours) we will check the negative number repository for corresponding request_id and will not create FDD.
Need to maintain negative file upload for reporting.
Check negative base using local cache when upload slot data.
3. Flow Execution
During execution on KP, we will perform all essential checks as currently practiced, including verifying the slot time. If the execution time falls outside the designated slot time, the task will be marked as a failure (out of slot time). We will not reschedule the task, even if reattempts and rescheduling within the next business hours are defined in the flow.
4. Reports
There will be no changes required in the approach for EOD and raw reports. Currently, EOD reports are generated based on the FlowExecutionAttemptDetailDocument start time. Raw reports are generated based on start time, phonon_uuid, request_id, and other relevant numbers.
Dashboard Report Count:
On page load data will be the current date.
Uploaded Record : Total record in file.
Attempted Records : Total FDD other then IN PROGRESS status.Attempted Records will be more then total record if multiple slots.
Negative Upload Records : All Negative Records uploaded for campaign.
Pending Records : All FDD count IN PROGRESS status.
Scheduled Timestamp : It will reflect the list of all slots timing if we have defined timestamp in dynamic slotting. In case , we have no timestamp in dynamic slotting , then it will take scheduled time of file upload.
Status : We will show the status of campaign as :-
Scheduled - If the last request ID was scheduled in file.
Rescheduled – If the file was rescheduled
Completed - If the last slot was completed.
Negative upload - If we have uploaded any negative upload in the campaign.
Yet to process : If we add Yet To process count in Campaign manager table page will take time to load because count will fetch from mongo FDD for each request id.
Campaign Details : All count will be fetch from mongo document it will be take time.
5. Retrigger
All FDDs with statuses such as IN PROGRESS, PARTIAL COMPLETE, etc., will be re-triggered for execution. This means all these FDDs will be placed in a queue for execution, and their scheduled time will be updated accordingly. A new flag will be added to the FDDs to indicate re-triggered records.
If re-trigger flag is true in FDD then we will not check slot time in execution. If re-trigger in custom slot then check custom slot.
If we have scheduled time in slots then retriggering is possible only after the scheduled time in the slot ends. And if there is no timestamp given in the slots then business hours will be counted and we can retrigger only after the business hour time ends. Retrigger will be disabled if the data is in processing . The retriggering will be enabled only the data processing is completed if the data is in progress then on next day (after business hours) we can retrigger.
NOTE : A separate service will be required specifically for this re-triggering feature.
While clicking delete button (which is for revoking campaign ) , it will change the status of campaign on UI. It will show cancelled instead or rescheduled etc.
0 Comments