Tasks and asynchronous operations

    Many operations in Meilisearch are processed asynchronously. These API requests are not handled immediately—instead, Meilisearch places them in a queue and processes them in the order they were received.

    Which operations are asynchronous?

    Every operation that might take a long time to be processed is handled asynchronously. Processing operations asynchronously allows Meilisearch to handle resource-intensive tasks without impacting search performance.

    Currently, these are Meilisearch's asynchronous operations:

    Understanding tasks

    When an API request triggers an asynchronous process, Meilisearch creates a task and places it in a task queue.

    Task objects

    Tasks are objects containing information that allow you to track their progress and troubleshoot problems when things go wrong.

    A task object includes data not present in the original request, such as when the request was enqueued, the type of request, and an error code when the task fails:

    {
        "uid": 1,
        "indexUid": "movies",
        "status": "enqueued",
        "type": "documentAdditionOrUpdate",
        "canceledBy": null,
        "details": { 
            "receivedDocuments": 67493,
            "indexedDocuments": null
        },
        "error": null,
        "duration": null,
        "enqueuedAt": "2021-08-10T14:29:17.000000Z",
        "startedAt": null,
        "finishedAt": null
    }
    

    For a comprehensive description of each task object field, consult the task API reference.

    Summarized task objects

    When you make an API request for an asynchronous operation, Meilisearch returns a summarized version of the full task object.

    {
      "taskUid": 0,
      "indexUid": "movies",
      "status": "enqueued",
      "type": "indexCreation",
      "enqueuedAt": "2021-08-11T09:25:53.000000Z"
    }
    

    Use the summarized task's taskUid to track the progress of a task.

    Task status

    Tasks always contain a field indicating the task's current status. This field has one of the following possible values:

    succeeded, failed, and canceled tasks are finished tasks. Meilisearch keeps them in the task database but has finished processing these tasks. It is possible to configure a webhook to notify external services when a task is finished.

    enqueued and processing tasks are unfinished tasks. Meilisearch is either processing them or will do so in the future.

    Global tasks

    Some task types are not associated with a particular index but apply to the entire instance. These tasks are called global tasks. Global tasks always display null for the indexUid field.

    Meilisearch considers the following task types as global:

    NOTE

    In a protected instance, your API key must have access to all indexes ("indexes": [*]) to view global tasks.

    Task queue

    After creating a task, Meilisearch places it in a queue. Enqueued tasks are processed one at a time, following the order in which they were requested.

    Task queue limit

    When the task queue reaches its limit (about 10GiB), it will throw a no_space_left_on_device error. Users will need to delete tasks using the delete tasks endpoint to continue write operations.

    Task queue priority

    Meilisearch considers certain tasks high-priority and always places them at the front of the queue.

    The following types of tasks are always processed as soon as possible:

    1. taskCancelation
    2. taskDeletion
    3. snapshotCreation
    4. dumpCreation

    All other tasks are processed in the order they were enqueued.

    Task workflow

    When you make a request for an asynchronous operation, Meilisearch processes all tasks following the same steps:

    1. Meilisearch creates a task, puts it in the task queue, and returns a summarized task object. Task status set to enqueued
    2. When your task reaches the front of the queue, Meilisearch begins working on it. Task status set to processing
    3. Meilisearch finishes the task. Status set to succeeded if task was successfully processed, or failed if there was an error
    NOTE

    Terminating a Meilisearch instance in the middle of an asynchronous operation is completely safe and will never adversely affect the database.

    Canceling tasks

    You can cancel a task while it is enqueued or processing by using the cancel tasks endpoint. Doing so changes a task's status to canceled.

    NOTE

    Tasks are not canceled when you terminate a Meilisearch instance. Meilisearch discards all progress made on processing tasks and resets them to enqueued. Task handling proceeds as normal once the instance is relaunched.

    Deleting tasks

    Finished tasks remain visible in the task list. To delete them manually, use the delete tasks route.

    Meilisearch stores up to 1M tasks in the task database. If enqueuing a new task would exceed this limit, Meilisearch automatically tries to delete the oldest 100K finished tasks. If there are no finished tasks in the database, Meilisearch does not delete anything and enqueues the new task as usual.

    Examples

    Suppose you add a new document to your instance using the add documents endpoint and receive a taskUid in response.

    When you query the get task endpoint using this value, you see that it has been enqueued:

    {
        "uid": 1,
        "indexUid": "movies",
        "status": "enqueued",
        "type": "documentAdditionOrUpdate",
        "canceledBy": null,
        "details": { 
            "receivedDocuments": 67493,
            "indexedDocuments": null
        },
        "error": null,
        "duration": null,
        "enqueuedAt": "2021-08-10T14:29:17.000000Z",
        "startedAt": null,
        "finishedAt": null
    }
    

    Later, you check the task's progress one more time. It was successfully processed and its status changed to succeeded:

    {
        "uid": 1,
        "indexUid": "movies",
        "status": "succeeded",
        "type": "documentAdditionOrUpdate",
        "canceledBy": null,
        "details": { 
                "receivedDocuments": 67493,
                "indexedDocuments": 67493
        },
        "error": null,
        "duration": "PT1S",
        "enqueuedAt": "2021-08-10T14:29:17.000000Z",
        "startedAt": "2021-08-10T14:29:18.000000Z",
        "finishedAt": "2021-08-10T14:29:19.000000Z"
    }
    

    Had the task failed, the response would have included a detailed error object:

    {
        "uid": 1,
        "indexUid": "movies",
        "status": "failed",
        "type": "documentAdditionOrUpdate",
        "canceledBy": null,
        "details": { 
                "receivedDocuments": 67493,
                "indexedDocuments": 0
        },
        "error": {
            "message": "Document does not have a `:primaryKey` attribute: `:documentRepresentation`.",
            "code": "internal",
            "type": "missing_document_id",
            "link": "https://docs.meilisearch.com/errors#missing-document-id"
        },
        "duration": "PT1S",
        "enqueuedAt": "2021-08-10T14:29:17.000000Z",
        "startedAt": "2021-08-10T14:29:18.000000Z",
        "finishedAt": "2021-08-10T14:29:19.000000Z"
    }
    

    If the task had been canceled while it was enqueued or processing, it would have the canceled status and a non-null value for the canceledBy field.

    After a task has been deleted, trying to access it returns a task_not_found error.