Do you want to learn Web Jobs and Azure Function in Depth and want to avoid pitfalls while doing development? If yes, then this article is for you.

We will look into Web Job and Azure function in depth and will create some Proof-Of-Concept projects. I will walk you through pitfalls of using different kinds of Web Jobs and Azure Functions bindings and what is suitable for which condition. This series would be long as we will discover and test both the framework with high load, Long running tasks and with different use-case.

Before diving in I want to inform you that Web Job comes under PaaS service while Azure Function comes under FaaS or LaaS popularly known as Serverless computing.

As Web Jobs comes under PaaS service offering, So lets get started with Web Jobs and once we are done with all bindings of Web Jobs we will than go and checkout Azure function binding in depth.

If we are using PaaS as our platform service to deploy our resources on Azure and we have requirement to Run Background tasks in our App Service Plan. We can use Web Jobs which enable us to Run Background tasks with the help of scripts or programs. You would not be charged any additional cost for running Web Jobs. But Web Jobs consumes resources of your App Service Plan. So if you have a Web app running on the same App Service Plan. It might happen that when your several Web Jobs are running, you might feel the slowness in your Web App because resources being used by the Web Jobs.

Special For You! Are you getting started as an Azure .NET developer and are not aware from where to start and when to use which service? Have you already checked out my Azure Book for .NET Developers ? It’s a full basic learning book getting started with Microsoft Azure Step by Step in 7 days for .NET Developers and many bonuses included source code and access to other Azure Topics.   Click here to check it out!.

An Era of Task Scheduler and Windows Services

Earlier on whenever there was a requirement to run any Background tasks we normally opt for Windows service or creating a console application and running that console application in Task Scheduler which will trigger Daily/ Weekly/ Monthly/ One time/ When computer starts/ When I log on / When a specific event is logged as shown in Figure 1. 

                                                              Figure: Era of Task Scheduler and Windows Service

As in Web App we don’t have any windows server allocated to us we can’t deploy or create our own windows service. So alternatively we can use Web Job to run our long running Tasks.

Types of Web Jobs

Continuous

• Always running Jobs.
• Starts immediately when deployed.
• Runs on all instances that the web app runs on

Triggered

• Runs on a Schedule or manually
• Runs on a single instance

Lets Talk about Triggered Web Jobs

Triggered Web Jobs are basically Background task which are executed or triggered based on the time schedule occurrence as we did in Task Scheduler and Windows Service. Triggered job can run for long running process but only single instance of Job will run.

Lets create and deploy Timer Trigger Web Job using Visual Studio on Azure.

Prerequisite:

• Visual Studio 2017
• Azure SDK installed
• Active Subcription on Azure portal
• Resources needed for this demo App Service Plan, Web for windows, Azure SQL and Azure Storage.

Create a new Project in Visual Studio 2017

 

By default when we create a Web Job project, it generates a Continuous running QueueTrigger binding template which we will check in Continuous Running Web jobs. Lets make some small changes to create Triggered Web Job.

Install below nuget dependencies

1. Microsoft.Azure.WebJobs.Extensions version=”2.2.0″

Once done change the QueueTrigger to TimmerTrigger.

Please donot update the Microsoft.Azure.WebJobs version to 3 else project will not build because in latest SDK JobHostConfiguration is obsolete. Please refer this article for detailed information Microsoft.Azure.WebJobs.3.0.2.

Let change the JobHost configuration to enable Timer Trigger job.

Great we have successfully configured the Timer Trigger template. Before going ahead, let’s talk about Cron expressions which are used to mention time recurrences in Timer Trigger Web Jobs.

public static void ProcessQueueMessage([TimerTrigger(“*/5 *  * * * *”, RunOnStartup = true)] TimerInfo timer, TextWriter log)

Over here */5 *  * * * * is a Cron expression which specifies the time at which this Timer Trigger Web Job will get executed. 
Cron definition as per wikipedia:

“The software utility cron is a time-based job scheduler in Unix-like computer operating systems. People who set up and maintain software environments use cron to schedule jobs (commands or shell scripts) to run periodically at fixed times, dates, or intervals. It typically automates system maintenance or administration—though its general-purpose nature makes it useful for things like downloading files from the Internet and downloading email at regular intervals”

To create cron expression as per your requirement and detailed information you can refer below websites:

• https://cronexpressiondescriptor.azurewebsites.net/
• https://crontab.guru/” target=”_blank”>

Let’s go to the https://cronexpressiondescriptor.azurewebsites.net/ website and create our cron expression to run a job every 1 hour.

Add Azure Storage connection string in app config as shown below:

 

Now lets run our Web Job  press F5 to check the time it is schedule to run.

The Web Job will run as per scheduled Time. If you want to run the job as soon as it’s deployed you can enable RunOnStartup =true as shown below:

public static void ProcessQueueMessage([TimerTrigger(“0 */60 * * * *”,RunOnStartup =true)] TimerInfo timer, TextWriter log)

 

Deploying Web Job on Azure

As Web Job is a part of Web App we deploy them as a part of Web App. In order to deploy the Web Job on Azure I will download the publish profile of my Web App on Azure.

 

Now Publish the Web Job from Visual Studio 2017

Set the Timer Trigger Job as Run On Demand

 

 

Once the Web Job has been deployed, Let’s go the Azure portal and enable AlwaysOn feature in Applicaion settings section which will not timeout our Web Job and then go to Web Job section in my Web App.

 

 

Click on Run to start the Web Job. Once the job is in running status click on logs to check the logs generated by Web Job. A new tab will open, now select job to check the logs of that Web Job.

 

 

Click on Running Status  to check the logs 

You can clearly see that our Job executed successfully and wrote rnng on console window.

Is this looks like a Real Life requirement Scenario ?

Yeah you guessed it right No, In Real life project we don’t have these kind of requirements. We might have other complex requirements like:

• Send Reports to Users who has subscribed to any Reports present in Web Application.
• Doing a Syncing DB operations to keep DB up to Date.
• Updating DB records based on certain logic.
• Refreshing the inventory records from DB and exporting it to a JSON file.
• Etc.

To make things real life, now let’s write a long running logic to be executed by Timer Trigger function. Considering that our logic will complete in 20-30 minutes and should be executed once a day I have changed the Cron expression to 0 20 4* * * i.e 4:30 am and i have enabled RunOnStartup to run the Web Job as soon as it’s deployed. 

Note*–  As a best practice use TextWriter log only for writing small information because this logger will not allow large number of logs.
In order to solve the limitation of logs in console windows in Web Jobs. We will implement custom logging with the help of log4net in this project and enable logging to a web job folder in Web Job deployment directory. So Now in this Use Case we will read the records from Azure SQL DB and write them to the log file. This job will be long running as we also want to check if we can run the long running Timer Trigger function or not?

Dependencies:

• log4net version: 2.0.8

Code Changes

Program.cs 

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using log4net;
using Microsoft.Azure.WebJobs;

namespace AzureTimerTrigger
{
    // To learn more about Microsoft Azure WebJobs SDK, please see https://go.microsoft.com/fwlink/?LinkID=320976
    class Program
    {
        // Please set the following connection strings in app.config for this WebJob to run:
        // AzureWebJobsDashboard and AzureWebJobsStorage
        public static ILog AlaLogger { get; set; }
        static void Main()
        {
            AlaLogger = LogManager.GetLogger("AzureTimerTrigger");
            var config = new JobHostConfiguration();

            if (config.IsDevelopment)
            {
                config.UseDevelopmentSettings();
            }
            config.UseTimers();
            var host = new JobHost(config);
            // The following code ensures that the WebJob will be running continuously
            host.RunAndBlock();
        }
    }
}

Functions.cs

using System;
using System.Collections.Generic;
using System.Configuration;
using System.Data.SqlClient;
using System.Diagnostics;
using System.IO;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using Microsoft.Azure.WebJobs;

namespace AzureTimerTrigger
{
    public class Functions
    {
        // This function will get triggered/executed when a new message is written 
        // on an Azure Queue called queue.
        private static readonly log4net.ILog Log = Program.AlaLogger;
        public static async void ProcessQueueMessage([TimerTrigger("0 */60 * * * *",RunOnStartup =true)] TimerInfo timer, TextWriter log)
        {
            log4net.Config.XmlConfigurator.Configure();
            Stopwatch sw = new Stopwatch();
            try
            {
                
                sw.Start();
                SqlConnection connection = new SqlConnection("DB Connection string");
                SqlCommand cmd = new SqlCommand();
                connection.Open();
                cmd.Connection = connection;
                cmd.CommandTimeout = 0;
                cmd.CommandType = System.Data.CommandType.StoredProcedure;
                cmd.CommandText = "sp_getAuthors";

                Log.Info("Executing sp");
                SqlDataReader reader = await cmd.ExecuteReaderAsync();
                while (await reader.ReadAsync())
                {
                    Log.Info(reader["Id"].ToString() + ", ");
                    Log.Info(reader["Author_name"].ToString() + ", ");
                    Log.Info(reader["country"].ToString() + ", ");

                }
                connection.Close();
                sw.Stop();
            }
            catch (Exception ex)
            {
                Log.Info(ex.Message);
            }
            Log.Info($"Completed Web job, Total time taken {sw.Elapsed.Minutes}");
        }
    }
}

Below is the stored procedure which I am calling from Web Job 

SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
ALTER procedure [dbo].[sp_getAuthors]      
as      
begin      
      
 WAITFOR DELAY '00:01:00'    
select  *  from tblAuthors      
end     

 

This table contains 490077 records. Just to create a scenario where the Job will taken more than 10 minutes to read these records. In order to be on safer side I have also added 1 minutes Wait in Store Procedure. With this I am trying to create a Real Life use case where we are reading DB table records and also to test Long running task on Web Job.

Add following Code for Log4Net configuration in appsetting.config after configuration

<configSections>
    <section name="log4net" type="log4net.Config.Log4NetConfigurationSectionHandler, log4net, Version=2.0.8.0, Culture=neutral, PublicKeyToken=669e0ddf0bb1aa2a" />
  </configSections>
  <log4net>
    <root>
      <level value="Debug" />
      <appender-ref ref="LogFileAppender" />
    </root>

    <logger name="ServiceBusTrigger">
      <level value="INFO" />
    </logger>

    <appender name="LogFileAppender" type="log4net.Appender.RollingFileAppender">
      <file value="D:\Article\YouDon'tKnowWebJobs\AzureTimerTrigger\AzureTimerTrigger\bin\Debug\Logs\logs.txt" />
      <appendToFile value="true" />
      <param name="AppendToFile" value="true" />
      <rollingStyle value="Size" />
      <maxSizeRollBackups value="10" />
      <maximumFileSize value="3MB" />
      <staticLogFileName value="true" />
      <layout type="log4net.Layout.PatternLayout">
        <param name="ConversionPattern" value="%date [%thread] %-5level %logger - %message%newline" />
      </layout>
    </appender>
    <logger name="SleepyCore">
      <level value="INFO" />
    </logger>
  </log4net>

Run the Web Job after making above changes to test everything is working fine. We can see in the Figure that Log has been created and we are able to communicate with Database which is present on Azure.

 

Once we have confirmed that everything is working fine. Now deploy the Web Job on Azure with steps that we followed earlier in this article.

Now our next task is to change the path of logs as per our deployment path on Azure. In PaaS Azure provide us D drive which is basically an Operating system drive. We should use the convenient faux absolute path D:\home\site that App Service provides us always. In order to check this path we will take help of KUDU.

 KUDU
  “KUDU is set of troubleshooting and analysis tools which provided with PaaS service offerings called KUDU. It is really helpul for capturing a memory dump, process explorer  looking at deployment logs with date time, viewing configuration files lile appSettings.”

If you want to learn more about KUDU, you can go and checkout this video by Scott Hanselman with David Ebbo.

https://www.youtube.com/watch?v=_fhmUqNGz2Y

If you are using PaaS serivce offerings of Azure you must know KUDU analysis tools to check the deployment summary, number of instances running etc. Now lets check how to go to KUDU tool and change our logs path step by step as shown below:

1. Go to web app on Azure portal
2. Go to Advance tool section and click on Go
   

           3. Click on Debug Console and select anyone of the option available eg.Cmd.
           4. Below window will open
           

          5. Go to “D:\home\site\wwwroot\app_data\jobs\triggered\AzureTimerTrigger>” path now 

  6. Edit AzureTimerTrigger.exe.config

    7. Save the file and now from Azure portal run the job manually.

 

8. Job Running

9. Now check the logs in the Logs folder that we configured in applicationSetting.

We can see that logs are getting generated. Now let’s wait for the job to complete. What do you think Long running Web Job will be completed without any errors or not? 

So after waiting for around 27 minutes our Web Job completed successfully:

So finally we have done a 1 round of testing of a long running background Timmer Trigger job and it has  successfully completed without any errors.

Lets do another load and increase the wait time to 2 minutes to see what happens when increase waiting time in Stored Procedure?

USE [TestDB]
GO
/****** Object:  StoredProcedure [dbo].[sp_getAuthors]    Script Date: 04/27/2019 16:48:53 ******/
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
ALTER procedure [dbo].[sp_getAuthors]      
as      
begin      
      
 WAITFOR DELAY '00:02:30'    
select  *  from tblAuthors      
end     

Now Run the Job Manually again and wait till the Job is completed or throws any Time Out exception or not?

After waiting for 52 minutes our Web Job completed successfully.

We can conclude that TimerTrigger jobs can be executed for long running Background Task, If the recurrences time of Web Job occurs then both the job will be running. But we have to ensure that  AlwaysOn mode  is enable because Continuous and scheduled CRON Web Jobs require ‘Always on’ to be enabled for your app.

 

WEBJOBS_IDLE_TIMEOUT:

In order to elaborate WEBJOBS_IDLE_TIMEOUT, Now lets see our Web Job section to check the state of Timer Trigger job once it’s completed and is idle for atleast 10 minutes.

You will see the status as Failed. Lets check the logs first to investigate this issue.

We can see that due to no output or CPU activity for 121 seconds our Web Job was timeout and it’s instructed to add SCM_COMMAND_IDLE_TIMEOUT or WEBJOBS_IDLE_TIMEOUT.

As our Web Job is recurrences after 1 hour we have to set WEBJOBS_IDLE_TIMEOUT to 60 minutes because we know max to max there would be 60 minutes time for our next run. So now add WEBJOBS_IDLE_TIMEOUT =3600 in application settings.

If you want to check other configuration setting related to Web Job you can checkout this article All about WebJobs. Lets run our Job manually and remove the timeout period from Stored procedure to make things faster and than manually run the job again and let it complete  and then check after sometime whether you get the timeout exception again or not?

You can see in the above figure that our Job first ran at 04/28/2019 11:19:04 and finished at 04/28/2019 11:44:29. Then Job was idle till 12:00:00, At 12:00 our job again triggered and completed at 04/28/2019 12:25:18.

So finally we saw Timer Trigger Job is working fine without any timeout exception, once we increased  WEBJOBS_IDLE_TIMEOUT. 

Key Takeaway:

•  We can run long running background job using Timer Trigger Web Jobs
•  What is KUDU, Cron Expressions
• Deploying Web Job on Azure
• WEBJOBS_IDLE_TIMEOUT

If you want to learn Azure Step by Step my honest suggestion would be to go and watch this video.