In our recent project, The user should be able to create a task with video file which associated with multiple audio/subtitle files.

A user can view that video and change audio track/subtitle. The user can also seek the video using subtitle transcription. There are other typical use cases where a user can edit the video/audio/subtitle, which would again need to process the whole thing!

We were not able play all these files together with HTML 5 video tag directly. So we needed to transcode the video file into some sort of stream supported in the video tag, so we can use it with the video tag.

HLS to the rescue!

Apple’s HLS(HTTP Live Streaming) stream was such a solution.

So, to transcode the video file into HLS stream we used FFmpeg multimedia framework.

FFmpeg is the leading multimedia framework, able to decode, encode, transcode, mux, demux, stream, filter and play pretty much anything that humans and machines have created.

It contains ffmpeg which can be used by end users for transcoding.

But It was not appropriate to make a user wait until we finished our stream transcoding.

In case of a large file, we were facing server timeout. So we moved transcoding part to the background process.

We used celery to perform background processing.

Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well. Tasks can execute asynchronously or synchronously.

We used redis as an in-memory database for celery, Which is an open source in-memory data structure store, used as a database, cache and message broker.

You can install celery by using the following command:

$ pip install celery

As I mentioned above we were using redis database in our project. You can install by executing following command from your shell:

$ pip install redis

To check whether redis is installed or not simply write following in your shell:

$ redis-server

We need to create a celery.py file in our project directory to use celery into our Django project.

Add the following code in your file and add the path to your settings and your app name.

To load our Celery app when Django starts, we need to import our app from celery.py file into __init__.py sits next to our settings file.

We need to add the following piece of code to use redis in our project:

BROKER_URL = 'redis://localhost:6379'
CELERY_RESULT_BACKEND = 'redis://localhost:6379'
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TIMEZONE = 'you can specify any timezone here'

You can specify your required timezone here.

You can receive celery task by running the following command:

$ celery -A 'your settings.py containing folder name' worker -l info

To run the celery scheduler run the following command:

$ celery -A 'your settings.py containing folder name' beat -l info

Celery scheduler will get all scheduled task and pass them to celery worker for processing.

Now just create a file task.py sits next to views.py or any other location. you can write your tasks in this file and call them from django views or simply run them by creating the schedule.

Here, you need to tell celery worker from where to get celery task. You can do this simply adding your tasks.py in your settings.py file.

CELERY_IMPORTS=("your_app_name.tasks")

Now we are done with celery setup in our Django project. Just see how to create celery task and periodic task. You can write your tasks in your tasks.py file.

You can call your celery task from your Django views by simply calling them in the following manner:

add.delay(7, 8)

Now We are done with background processing with Django using Celery and Redis.

Click here for more details…

At BoTree Technologies, we build enterprise applications with our Django team of 20+ engineers.

We also specialize in RPA, AI, Python, Ruby on Rails, JavaScript and ReactJS.

Consulting is free – let us help you grow!