Storage Limits

Because the FeedMaster node has to queue up requests, a certain amount of storage is required. The requests stay in the queue and take up space on disk until the message has been processed by the pipeline.

  • The default space for queued messages is 10GB.
  • The maximum storage which can be configured depends on the disk space available.
  • As the storage limit is reached, the FeedMaster rejects new requests by failing with HTTP 420 error message, Rate limit exceeded.
  • If one of the upstream services used by the pipeline is slow, that can cause the queue to grow. Setting the storage limit higher allows for more requests to be queued up, but the higher setting also delays the clients from being notified about the upstream slowness.
  • The storage space used is for each FeedMaster node. If your deployment consists of multiple FeedMasters, each request takes up space on one of the FeedMaster nodes.
  • All Ultra Tasks configured on a Snaplex share this disk space. The jcc.broker.disk_limit property specifies the maximum space used to store queued messages on disk when the message is being processed. The default value is 10GB.
    • If your Snaplex uses the slpropz configuration method, you can configure this value in Snaplex > Node Properties in Manager by specifying the limit in bytes in the Global properties field set:

      jcc.broker.disk_limit=20971520

    • If your Snaplex does not use the slpropz configuration method, you must add the jcc.broker.disk_limit value as an entry in the global.properties configuration file.
    • You can also change the storage location for the broker files using the following properties:

      jcc.broker_data_dir=/mnt/data

      jcc.broker_sched_dir=/mnt/data

      jcc.broker_tmp_dir=/mnt/data

  • The default location is $SL_ROOT/run/broker. Ensure that the mount you use for the broker files has sufficient disk space as per the disk_limit configuration. The snapuser user should have full access to the data location
  • You can also change the storage location for the broker files using the following properties:

    jcc.broker_data_dir=/mnt/data

    jcc.broker_sched_dir=/mnt/data

    jcc.broker_tmp_dir=/mnt/data

  • The default location is $SL_ROOT/run/broker. Ensure that the mount you use for the broker files has sufficient disk space as per the disk_limit configuration. The snapuser user should have full access to the data location.
Note:

When running multiple Tasks on one Snaplex/FeedMaster, consider the following:

  • If a Task is running slow, the FeedMaster starts queueing the requests, and clients sending the requests begin receiving errors. Other Tasks on the Snaplex should not be affected.
  • If more than half of the Tasks running on the Snaplex are slow, then the requests to the slow Tasks begin receiving errors. But the other Tasks should not be affected.

Sometimes, all Tasks may be affected, even though some isolation between Task performance is maintained by the FeedMaster. To ensure complete isolation, it is recommended that you use separate Snaplex instances for different workloads and to configure enough space. Thus making it easier to meet your SLA (Service Level Agreements).

Timeout between FeedMaster and clients

The timeout between the FeedMaster and client triggering requests to Ultra Tasks is user configurable. You can configure the timeout on a per-request basis by passing an X-SL-RequestTimeout header with the timeout value stated as the number of milliseconds (the default is 900000). You can also configure the timeout for the FeedMaster as a whole with the jcc.llfeed.request_timeout global property. Ultra requests are guaranteed to run at least once before the timeout.