How to deploy Custom Libraries in AWS Lambda Layers

In the previous article, we have looked in detail about AWS Lambda Layers and have also seen how we can make use of Layers to reduce the deployment package size in our Lambda API project by pushing all the default package references into a “dependenciesLayer” we created.

This setup works well for simple APIs which are limited to a single library which contains the actual Lambda function. But if we move towards functions which are designed using a layered architecture, with each layer taking up a specific responsibility such as Business, Domain, Infrastructure and so on, we are left with two options while deploying such functions into Lambda:

1. Deploy the lambda function along with binaries of these custom libraries which the function depends in the same package.
2. Deploy the lambda function alone with a dependency library, which looks up in a Layer that contains the binaries of these custom libraries everytime required.

While the first option is something which we generally do subconsciously, the second option is something which can offer us a few advantages such as:

1. Update the custom libraries without having to touch the lambda
2. Reuse these custom libraries across different lambda services wherever required
3. The deployment size of the lambda functions would decrease making them easier to deploy

In this article, let’s look at how we can deploy a custom library which contains our own business logic into a lambda layer and link it to our lambda function for use.

Let’s assume we have an API /AwsLayers.App which exposes an endpoint /api/values that internally calls a function from a class library AwsLayers.Util project. This class library AwsLayers.Util has also an S3Utility class that exposes methods to access S3 bucket contents. This makes the class library depend on a few packages like the AWSSDK.S3 and so on. Now we’re interested in pushing this class library into a Layer so that it can be reused in other lambda functions as well.

We achieve this in four steps:

1. Pack the custom library along with its dependencies into a Nuget Package
2. Refer the dependenciesLayer project to the Nuget package created, which acts as the runtime store manifest.
3. Refer the API project to the dependenciesLayer project and push all the package references in the API project to the dependenciesLayer project. This way lambda function would’nt contain any other package references other than the dependenciesLayer itself
4. Create runtime package store from the dependenciesLayer and deploy it to a Layer. Use this layer while creating the deployment package of the Lambda function from the API project.

The /values endpoint of the API project looks like below:

[Route("api/[controller]")]
public class ValuesController : ControllerBase
{
  private readonly IValuesUtility utility;

  public ValuesController(IValuesUtility utility)
  {
    this.utility = utility;
  }

  // GET api/values
  [HttpGet]
  public IEnumerable<Value> Get()
  {
    return this.utility.Seed();
  }
}

The IValuesUtility class comes from the AwsLayers.Util library, which is as follows:

public interface IValuesUtility
{
  IEnumerable<Value> Seed();
}

public class ValuesUtility : IValuesUtility
{
  public IEnumerable<Value> Seed()
  {
    var values = new List<Value>();

    for (int i = 1000; i < 1100; i++)
    {
      values.Add(new Value
      {
        Id = Guid.NewGuid(),
          ItemValue = i*6
      });
    }

    return values;
  }
}

The csproj of the AwsLayers.Util library looks like below:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>netcoreapp3.1</TargetFramework>
    <PreserveCompilationContext>true</PreserveCompilationContext>
    <GenerateRuntimeConfigurationFiles>true</GenerateRuntimeConfigurationFiles>
    <AWSProjectType>Lambda</AWSProjectType>
    <OutputType>Library</OutputType>
    <StartupObject />
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="AWSSDK.S3" Version="3.3.110.58" />
    <PackageReference Include="Microsoft.Extensions.Configuration.Abstractions" Version="3.1.5" />
    <PackageReference Include="Microsoft.Extensions.Logging.Abstractions" Version="3.1.5" />
  </ItemGroup>
</Project>

To deploy AwsLayers.Util library, let’s begin by packing this library into a Nuget package. To do this, we use the ‘dotnet pack’ command which we run inside the Util directory as:

> dotnet pack -c:Release -p:PackageVersion=1.4.0

It packs the class library along with all its dependency into a nuget package and produces a nupkg under /bin/release. The PackageVersion refers to the version of the nupkg generated: it is any arbitary value we provide to specify that this version of the package has been created and referenced to.

Next, we need to instruct our NuGet package manager to use this nupkg when referenced. To do this, we have to create a “Local Nuget Store” that contains all the Nuget packages created locally and then add this value to Nuget Sources.

To do this, we can either use Visual Studio to edit the Nuget Sources by clicking on Nuget Package Manager => Options => Package Sources and add our value, or directly edit the Nuget.Config file which contains the list of NugetSources to look into. The file is located at %appdata%NuGetNuGet.Config (Windows) and ~/.nuget/NuGet/NuGet.Config (Mac/Linux).

Copy the nupkg file we just created into a common folder which we shall treat as a “Local Nuget Store” and then open the Nuget.Config file to add this path in the section.

<packageSources>
  <add key="nuget.org" value="https://api.nuget.org/v3/index.json" protocolVersion="3" />
  <!-- Local Nuget Store -->
  <add key="LocalNuget" value="E:LocalNuget" />
  .. other sources ..
</packageSources>

Once this is done, add a PackageReference in the dependenciesLayer project to the Util package we just created. The dependenciesLayer now looks like this:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>netcoreapp3.1</TargetFramework>
    <PreserveCompilationContext>true</PreserveCompilationContext>
    <GenerateRuntimeConfigurationFiles>true</GenerateRuntimeConfigurationFiles>
    <AWSProjectType>Lambda</AWSProjectType>
    <OutputType>Library</OutputType>
    <StartupObject />
  </PropertyGroup>
  
  <ItemGroup>
    <PackageReference Include="Amazon.Lambda.AspNetCoreServer" Version="5.1.0" />
    <PackageReference Include="AwsLayers.Util" Version="1.4.0" />
  </ItemGroup>
</Project>

Once added, just to verify run dotnet publish on the dependenciesLayer project and check if the Util library is now available in the bin folder.

> dotnet publish

Finally, add a project reference to this dependenciesLayer inside the App project as below:

<Project Sdk="Microsoft.NET.Sdk.Web">
  <PropertyGroup>
    <TargetFramework>netcoreapp3.1</TargetFramework>
    <GenerateRuntimeConfigurationFiles>true</GenerateRuntimeConfigurationFiles>
    <AWSProjectType>Lambda</AWSProjectType>
  </PropertyGroup>
  <ItemGroup>
    <ProjectReference Include="..AwsLayers.DependenciesAwsLayers.Dependencies.csproj" />
  </ItemGroup>
</Project>

The App project doesn’t contain any of the packages it requires and instead it sources them from the dependenciesLayer which now contains all the packages which the App requires.

Create a package store using the dependenciesLayer csproj as the manifest as below:

> dotnet store --manifest AwsLayers.Dependencies.csproj --runtime linux-x64 --framework netcoreapp3.1 --framework-version 3.1.0 --output bin/dotnetcore/store --skip-optimization

It creates a package store and an artifact.xml which contains all the dependencies that are now available in the store. If everything goes well, we should now find the AwsLayers.Util dll in the store folder and its entry in the artifact.xml.

<StoreArtifacts>
  <Package Id="Amazon.Lambda.APIGatewayEvents" Version="2.0.0" />
  <Package Id="Amazon.Lambda.ApplicationLoadBalancerEvents" Version="2.0.0" />
  <Package Id="Amazon.Lambda.AspNetCoreServer" Version="5.1.0" />
  <Package Id="Amazon.Lambda.Core" Version="1.1.0" />
  <Package Id="Amazon.Lambda.Logging.AspNetCore" Version="3.0.0" />
  <Package Id="Amazon.Lambda.Serialization.SystemTextJson" Version="2.0.0" />
  <Package Id="Microsoft.Extensions.Configuration" Version="2.1.0" />
  <Package Id="Microsoft.Extensions.Configuration.Abstractions" Version="2.1.0" />
  <Package Id="Microsoft.Extensions.Configuration.Binder" Version="2.1.0" />
  <Package Id="Microsoft.Extensions.DependencyInjection.Abstractions" Version="2.1.0" />
  <Package Id="Microsoft.Extensions.Logging" Version="2.1.0" />
  <Package Id="Microsoft.Extensions.Logging.Abstractions" Version="2.1.0" />
  <Package Id="Microsoft.Extensions.Options" Version="2.1.0" />
  <Package Id="Microsoft.Extensions.Primitives" Version="2.1.0" />
  <Package Id="System.Runtime.CompilerServices.Unsafe" Version="4.5.0" />
  <Package Id="AwsLayers.Util" Version="1.4.0" />
  <Package Id="AWSSDK.Core" Version="3.3.106.12" />
  <Package Id="AWSSDK.S3" Version="3.3.110.58" />
  <Package Id="Microsoft.Extensions.Configuration.Abstractions" Version="3.1.5" />
  <Package Id="Microsoft.Extensions.Logging.Abstractions" Version="3.1.5" />
  <Package Id="Microsoft.Extensions.Primitives" Version="3.1.5" />
</StoreArtifacts>

zip the dotnetcore folder into package.zip and upload it to S3, similar to how we did in our getting started article. Use the paths of the package.zip and the artifact.xml uploaded in the S3 bucket and create a new AWS Lambda Layer.

Copy the layerArn available in the top right of the Layer info page and note it down somewhere – as we shall now move on to our final step.

Back in our App project, let’s test how this layer impacts our deployment package by running the lambda package command twice:

#first time#

> dotnet lambda package 

#note the size of the package - its around 659KB#

#second time#

> dotnet lambda package --function-layers layerArnYouCopiedJustNow

#note the size of the package - its around 58KB#

Observe the log printed on the screen for both the command versions, the one with the function-layer specified makes it clear that there are many packages that are ignored in the deployment package because they’re already available in the Layer. This also includes the Util library that the deployment package skips because its there in the Layer we deployed before.

Finally, to test this let’s deploy the API project and see how this works. The sample project uses a serverless template from dotnetcore set of templates provided by AWS Toolkit. In the serverless.template file, look for the below section “Resources:AspNetCoreFunction:Properties” and add a property “Layers” which is a string array containing the layerArn we kept aside before.

"Resources": {
    "AspNetCoreFunction": {
        "Type": "AWS::Serverless::Function",
        "Properties": {
          "Runtime": "dotnetcore3.1",
          "Layers": [
              "arn:aws:lambda:us-west-2:123456789:layer:dependenciesLayer:5"
          ],
          ... other settings

Once this is done, run:

> dotnet lambda deploy-serverless

This command picks up the serverless.template along with other files and deploys our lambda function along with other components in the form of a cloudformation stack.

If everything goes well, you should now see the output in the command window with ApiUrl which we shall invoke with our /api/values to see if everything works well.

GET /Prod/api/values
HOST abcdefg.execute-api.us-west-2.amazonaws.com

Which now returns a bulk of records. In this way, we can deploy a custom class library which contains our own shareable business logic into lambda layers and configure our lambda function to access it.

The complete project used in this article is available at https://github.com/referbruv/aspnetcore-lambda-layers-example