.NET Core and System.IO.FileLoadException when deployed

If you see a System.IO.FileLoadException like so during runtime after deployment:

System.IO.FileLoadException: Could not load file or assembly 'Microsoft.Extensions.Hosting.Abstractions, Version=3.1.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. The located assembly's manifest definition does not match the assembly reference. (0x80131040)
File name: 'Microsoft.Extensions.Hosting.Abstractions, Version=3.1.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'
   at Microsoft.AspNetCore.Hosting.WebHostBuilder..ctor()

The deployment may be a docker container, execution of dotnet.exe with .dll entry point, a .NET exe. This can be caused by what the exception says, file not found, or it could be that publish was not used to when creating the deployment bits.

dotnet publish "your_project.csproj" -c Release -o /publish

Note, that just using dotnet build and using the output may actually not lead to this exception during runtime. I've just seen it occur in Docker and running off the output for a Azure Service App. Oddly the same output used for an Azure Function does not throw this.

Ideally, I'd like to run dotnet restore, dotnet build (with no restore), dotnet test, and dotnet publish (--no-build) in this sequence in Azure DevOps build pipeline, but having the --no-build argument in dotnet publish causes the same exception above. Same issue when following the same steps in a Dockerfile. I suspect that some metadata must be generated on publish so just copying over the built output isn't enough. In the end, I just run the dotnet publish without the --no-build flag.

# Multistage image build

# Set environment variables and expose ports.
FROM mcr.microsoft.com/dotnet/aspnet:3.1-buster-slim AS base
LABEL author=YOUR_MOM
WORKDIR /app

# Kestrel default http port is 5000
ENV ASPNETCORE_URLS=http://+5000

EXPOSE 80 5000

COPY /src .
COPY /NuGet.Config .
WORKDIR /src/API

# Handle your own authentication to private artifact feeds through Nuget.Config or Credential Provider.
RUN dotnet restore "YOUR_PROJECT.csproj"
RUN dotnet build "YOUR_PROJECT.csproj" --no-restore -c Release -o /app/build

# Publish with all dependencies to output directory.  
# Ideally we want a multistage build where we restore, build, test, then publish.  
# Publish may output different bits depending on passed in input (project or solution - different versions from other projects).
FROM build AS publish

# If we use --no-build the there is a problem with the publish-time manifest not picking up the correct version of dependencies.
RUN dotnet publish "BillingAddress.Api.csproj" -c Release -o /app/publish
#COPY --from=build app/build app/publish

# Build runtime image
FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "YOUR_PROJECT.dll"]