Problem
We manage a MISP instance which receives over 130,000 IOCs every day from multiple sources which we need to push to over 22 Microsoft Sentinel instances which we manage.
No matter how hard we have tried to get the Cudeso generated python scripts to push this volume we have hit issue after issue. These scripts are great if you have 1 or 2 Sentinel instances and average volumes of IOCs but struggle at scale.
There is no point trying to push 130,000 IOCs to all the instances at the same moment, so we have spread them out. Our worst feed however is responsible for 60,000 of the IOCs being in a single MISP event, and trying to push this single feed to our Sentinel instances also proved too problematic.
Our access tokens we generated from https://login.microsoftonline.com would expire long before we could push our IOCs to the Sentinel threat intelligence api (https://sentinelus.azure-api.net/{workspaceId}/threatintelligence:upload-indicators?api-version=2022-07-01).
When the python scripts would error out it was hard to troubleshoot where the error had occurred.
The Sentinel api has rate limit restrictions of only 100 IOCs per POST and 100 POSTs per minute. This means we have to send 28,000 POSTS per day to the api to load all our IOCs in the various instances.
All the above issues resulted in every Sentinel instance importing different numbers of IOCs where we wanted standardisation across them. Some would import 3,000, some 20,000 and some 60,000.
Solution
First and foremost, we spread out the IOC feeds we pull into our MISP platform throughout a 24hr period. As mentioned above this means the most we tackle in any one hour period is having to push 60,000 IOCs to Sentinel. We had some smaller IOC feeds which we pulled multiple times per hour but these did not pose a volume issue.
We then implemented NIFI to do the heavy lifting…….
We implemented multiple InvokeHTTP processors to call the Sentinel api load balanced via a DistributeLoad processor. We set the thread count on this InvokeHTTP processors to push the Sentinel API rate limit boundaries as much as possible to ensure the highest throughput and to ensure the various Sentinel instances all received the same IOCs at the same time. We also had to set retry logic for POSTs which failed due to the rate limit being struck.
