LangFlow on AWS EC2

by | Jun 2, 2023 | Software

LangFlow is a React front end for LangChain that makes it easier to visualize what you are doing. My team is currently using for a hackathon so we wanted a hosted version we could all play with.

I don’t know of any security holes in LangFlow, but it’s always safest to assume there are, so act accordingly.

I’ll add I first tried to get this running on Replit and completely gave up during the install process. We assume it’s an issue with NixOS. I’ll note it specifically broke trying to install llama-cpp-python. There is a previous version deployed to Replit, but even forking that and trying to upgrade did not work.

So I spun up an EC2 t3.small to install LangFlow but ran into problems. The first is that the AWS defaults do not offer enough disk space. Worse, no swap disk is configured and for some reason when pip moves files it puts them through memory, so installing a larger package like torch will kill your process due to OOM, and your connection, losing history, etc. It’s not super obvious this is the cause. My first time, I had to restart the whole instance.

Easy mode would be to just run a bigger machine, maybe t3.medium 4GB RAM would do it. But I kept at it, recreating the instance with more disk space (16GB) and a second volume for swap (2GB).

Referring to:
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-store-swap-volumes.html
https://stackoverflow.com/questions/74515846/error-could-not-install-packages-due-to-an-oserror-errno-28-no-space-left-on

$ swapon -s

Probably shows nothing, but some instance types are automatically configured with a swap partition or file.

$ lsblk -p
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
/dev/nvme0n1 259:0 0 16G 0 disk
/dev/nvme0n1p1 259:2 0 16G 0 part /
/dev/nvme0n1p127 259:3 0 1M 0 part
/dev/nvme0n1p128 259:4 0 10M 0 part
/dev/nvme1n1 259:1 0 2G 0 disk

That last one at 2G is the volume we provisioned or swap. To get the real name, you have to go to the storage tab of your EC2 dashboard, or try:

$ sudo /sbin/ebsnvme-id /dev/nvme1n1
Volume ID: vol-04d10a66ad2521d93sdb

Should match what the EC2 dashboard when you created the partition. Now create and verify the swap:

$ sudo mkswap /dev/sdb
Setting up swapspace version 1, size = 2 GiB (2147479552 bytes)
no label, UUID=2aff5f50-ff55-431e-a375-d2ea9edde8d9
$ sudo swapon /dev/sdb
$ free -h
total used free shared buff/cache available
Mem: 1.9Gi 189Mi 1.4Gi 0.0Ki 258Mi 1.5Gi
Swap: 2.0Gi 0B 2.0Gi

OK, now for the real show. Python 3 is installed, check with:

python3 –version

We can add an alias:

alias python='python3'

PIP is not installed. Followed instructions: https://pip.pypa.io/en/stable/installation/

But don’t use ensurepip because it won’t install a pip command – you’ll have to run

python -m pip …

every time. Yes, it can be aliased, but this solves it:

curl -O https://bootstrap.pypa.io/get-pip.pypython get-pip.py

Check with:

pip –version

Several packages are required, shown here in order of the installation breaking:

$ sudo yum install cmake
$ sudo yum install gcc
$ sudo yum install gcc-c++
$ sudo yum install python3-devel

 

$ pip install langflow

LangFlow is now installed and you can run it with the langflow command. However, it will bind to 127.0.0.1, preventing outside access. However, this is OK because we want to lock it down. If you run it like that, nothing is encrypted, including your API keys.

We have a couple choices here. The easy way is to set up an Application Load Balancer  in AWS, let it handle certs, etc. I’m pretty sure you can point it to the instance so if it changes, it automatically updates the IP. You get a load balancer in the free tier, too.

But I’m saving that for another project, so let’s got the cheapskate route and make it hard on ourselves. We’ll set up Nginx as a reverse proxy with Let’s Encrypt for certs, but you could use Apache HTTPD or HAProxy if you prefer.

sudo yum install nginx

Now it’s installed, you need to add a block in the config file for Certbox to update for you. I took the extra step to point a (sub)domain I own to the public IP of my EC2 box, which is the server name below. OK, full disclosure, I do not own example.com. But you get the picture.

$ nano /etc/nginx/nginx.conf

Add:

    server {
        server_name  langflow.madeupname.com;
        #root         /usr/share/nginx/html;

        # Load configuration files for the default server block.
        include /etc/nginx/default.d/*.conf;

        location / {
                proxy_pass http://127.0.0.1:7860;
                proxy_http_version 1.1;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection 'upgrade';
                proxy_set_header Host $host:$server_port;
                proxy_cache_bypass $http_upgrade;
        }
    }

Next step is getting a cert from Let’s Encrypt. It recommends installing Certbot via snapd. Snapd could be installed via yum/dnf if you had support for EPEL on AL2023, but they removed EPEL support because they don’t like you.1 So you have to install it via pip, which thankfully you just installed.

Instructions: https://certbot.eff.org/instructions?ws=nginx&os=pip

sudo dnf install augeas-libs
yum search certbot # verify you don’t have this installed already
sudo /opt/certbot/bin/pip install --upgrade pip
sudo /opt/certbot/bin/pip install certbot certbot-nginx
sudo ln -s /opt/certbot/bin/certbot /usr/bin/certbot
sudo certbot --nginx

That last step automatically updates nginx.conf for you and reloads, which is pretty slick.

To be extra safe, I created a langflow user to run this instead of ec2-user, since it won’t even have sudo access.

$ sudo adduser langflow
$ sudo cp -pr .local/ logs/ .config/ tmp/ .cache/ .chroma/ /home/langflow
$ sudo chown -R langflow:langflow /home/langflow
$ sudo su - langflow
$ langflow

Finally, go to the EC2 dashboard, Security tab, and add a new inbound rule – or change the existing one for HTTPS – to limit it to “My IP” so it’s only accessible by your box. At this point, you should be able to go to langflow.example.com and it should come up.

Good luck!

  1. Sorry, it’s possible they don’t like anybody. [↩]

samkgg Lessons Learned

by | Sep 15, 2022 | Software

(This is a regularly updated page where I document what I’ve learned while building samkgg, a demo app for AWS SAM, Kotlin, GraalVM, Gradle, and other technologies as this progresses.

Read the samkgg blog post first to understand what I’m doing. You can find the working code here:

https://github.com/madeupname/samkgg

Builds and Dependencies

It’s important to consider how you organize your code. There are a couple options that give you flexibility. A critical point is that you don’t need to use Graal for everything! The main advantage of Graal is fast cold startups. If you are serving users interactively, use Graal. But some Lambda functions run asynchronously or in batch and it doesn’t matter much then. It might save you developer time to build a standard JVM function and not worry about reflection. SAM supports this.

When you create your project with “sam init” as described in the README, it will create a single Lambda function in a single Gradle project. You can put multiple Lambda functions in that same directory and build/deploy them without error by configuring the path for each function in template.yaml. However, each function will have the same executable and hence the combined dependencies for all functions in your project. This is a little better with Graal, but not recommended.

There is no way to make this a traditional Gradle multi-project build. Each function needs its own Gradle build:

https://github.com/aws/aws-sam-cli/issues/3227

In lieu of a multi-project build, you can do two things. First is to follow Gradle guidelines for large projects. This is definitely not a large project, but we organize it like one. Second is to add more build logic in via Makefiles. Yes, Makefiles:

https://makefiletutorial.com

Other notes:

  • Despite the GraalVM term “native image” the SAM package type is still zip. Hence, Lambda layers are allowed.
  • Similarly, the use-container flag passed to “sam build” means it is going to build the functions inside a Docker container. Since it’s building a binary for a specific platform, the build runs in that target platform with the necessary dependencies.
  • The Kotlin stdlib JDK 8 dependency threw me, but that’s the latest version. It still works with JVM 17 targets.
  • AWS SDK v2 is recommended, most importantly since they are claiming it is GraalVM-safe. It was also reworked for faster Lambda cold starts. However, it is possible to include both v1 and v2 in the same project, which could be required given v2 still has not reached feature parity.

 

Windows

You want to run WSL 2 to support Docker and run Ubuntu locally (more below). This is possibly just my machine, but Docker Desktop makes the “WMI Provider Host” process suck up CPU, blasting the fans, and none of the solutions I’ve read fix it. YMMV.

Git Bash (MINGW64) seems to support the most commands well and doesn’t need Docker, so that’s my go-to shell. Of course, IntelliJ can run Gradle directly and that’s handy, too.

However, Ubuntu is required for running the tracing agent, which is required to ensure you have all your GVM resource and reflection config. And unfortunately that needs Docker Desktop running.

GraalVM

Note: I believe you can/should always use the latest stable version because you are creating an actual binary in a custom runtime. Even though you choose a GraalVM JDK version (11 or 17) when creating the app via the CLI, there’s no separate VM/runtime supplied by AWS. Native images are self-contained.

Minimum docs to read as of this writing. I’ve seen docs change significantly between versions.

https://www.graalvm.org/22.2/reference-manual/native-image/

https://www.graalvm.org/22.2/reference-manual/native-image/basics/

https://www.graalvm.org/22.2/reference-manual/native-image/metadata/

Skipping the docs will lead to a bad time. It may also lead to ignoring viable libraries just because they use reflection.

GraalVM can use code with reflection, it just needs to know about it beforehand. In particular, it needs to know all of the possibilities in advance. If your code dynamically instantiates any of number classes, Graal needs to know which classes those are so they are included in the executable.

Reachability Metadata and the Tracing Agent

One of the biggest challenges with working with Graal is configuring it to include what will be needed at runtime, including dynamic features and resources. native-image does its best to discover this at build time, but it can’t predict everything. Hence you must give it supplemental instructions via “reachability metadata” commonly shortened to metadata. Key files are:

  • resource-config.json – this forces the build tool to include these files so that they may be loaded at run time when it’s not obvious to the tool at compile time
  • reflect-config.json – specify what will be called reflectively so the VM can prepare this code for execution in the binary

Some libraries supply or document this metadata, but most haven’t. This is somewhat eased by using the Tracing Agent (see metadata docs above), which is a runtime Java agent that tracks every time reflection or dynamic behavior is used and stores everything in special config files. The two key ones are

However, the agent can only detect these usages during the run. If your run with the agent skips any classes, methods, even conditional branches, and they would have used reflection, the config files will not get updated and your code can have an error during runtime. A solution to this is to run the agent when you run your tests, assuming your tests have good coverage. You want to add exclude filters for your test libraries.

When I ran it on this simple project, though, the output was enormous (41KB) because it’s listing everything individually. It’s like importing every single class from a package instead of using a wildcard. A small binary is a high priority for Graal. The good news is there is a flag to merge metadata from multiple runs into a single file.

Given all that, you can see why all libraries and frameworks seeking to be GraalVM-friendly (like Micronaut and Quarkus) seeks to avoid reflection like the plague. Sadly, I was using Spock for tests (huge Groovy fan), and discovered the agent will include everything for Groovy and Spock, which is way too much to wade through. I then understood why plain old JUnit was chosen for the template.

Environments/Stacks

Environment is an overloaded term in SAM. You have:

  • SAM CLI environment
    • the “sam” command has a config file named samconfig.toml and this file divides configuration settings among environments
    • the default environment is literally called default; create others as needed
    • you specify the environment for the command with –config-file
  • environment variables
    • there is a block for this in template.yaml
  • environments where the function runs
    • local or AWS

Finally, what a programmer might think of as a deployed environment (qa, production) CloudFormation (and SAM by extension) calls a stack.

Per SAM best practices, you create a stack per environment, such as development, staging, or production. So together with environments, your code can be:

  • deployed to AWS in different stacks
    • development
    • production
  • local
    • Docker container (sam local invoke)
      • development
      • production
    • test (gradle test, no container)

Each of these has subtle differences that are not always obvious/documented. I do my best to document the surprises here.

SAM CLI can be passed a JSON file and/or command line options that contain overrides for the Environment variables in your template.yaml. Two critical points:

  • You have to create environment variables in template.yaml or the Lambda environment won’t pass them to your function, even if they exist.
  • One very misleading issue is that the Parameter block of the JSON file is for global environment variables, not parameters! I was not getting my global env vars overriden even though they were declared in template.yaml and specified for the function. For safety’s sake, I duplicate them in the JSON file.

Logging

I’m not the first engineer to find logging with Lambda and Graal to be surprisingly challenging. My initial choice for logging was AWS Lambda Powertools. It looks like a solid library to help you troubleshoot, however, it relies on Log4j 2 which is not GraalVM friendly. According to that thread, the maintainers say it won’t be ready for Graal until Log4j 3 at the earliest.

Graal officially supports java.util.logging (AKA JUL), which is nice because it’s one less dependency (setup docs here). However, for reasons I don’t yet understand, log messages directly from JUL did not show up when run from Lambda. They worked fine when testing outside the container, which you’ll find is a common challenge with this stack.

The solution was adding SLF4J with the JDK 1.4 logging support library (not the “bridge” library – that sends all JUL logs to SLF4J). SLF4J should also enable logging from AWS Java SDK, and I have seen an instance of an AWS SDK log message in my console.

The next challenge was determining per-environment configuration given the differences:

  • development and prod should have different logging.properties files to set different log levels
  • code deployed to AWS uses CloudWatch

My first attempt was to configure the loggers at build time so Graal didn’t have to include the files in the native image. But Graal considers creating an input stream from a resource to be unsafe during initialization. Logging is still configured during class initialization, but at runtime, not build time. I use AWS-provided environment variables to determine if it’s in a container and if that container is AWS-hosted.

The next issue is that Lambda logs everything in the console to CloudWatch, which is good, but CloudWatch sees every newline as the start of a new log message. I have created a CloudWatchFormatter replaces newlines with carriage returns (which CW doesn’t mind) if the code is running on AWS. My next goal is creating a JSON formatter to allow better use in CloudWatch Insights.

Another interesting concept is Mapped Diagnostic Context (MDC), which is like a thread-bound map. It is not supported by JUL, but SLF4J offers a basic adapter. You can put any key/value into the map and it will be visible to all methods that get the MDC adapter. I added the AWS Request ID from the context so it can be logged with messages from any source.

I owe much of this to Frank Afriat’s blog post and GitHub project. You may wish to use his implementation, which is more robust than mine, but is also marked alpha and relies on SLF4J’s simple logger, which is not as robust as JUL.

DynamoDB

I found the DynamoDB Guide an excellent supplement to the docs.

I don’t see any library that handles schema and data migrations for DDB like Flyway and Liquibase do for SQL. CloudFormation can build tables for you, but it can’t handle data migrations that naturally occur in an active project with a changing schema. Luckily, you can implement the basics of Flyway pretty trivially.

A particular challenge not shown by the docs is a very common scenario: a class has a list of custom objects. Mapping this in the enhanced client (their ORM) is not at all obvious. You have to map it to a list of JSON document, possibly with a custom attribute converter. I think they didn’t bother making this easy because with DDB, this relationship is only useful when it’s a composition relationship, meaning the objects in the collection don’t exist outside their parent. Otherwise, they would be stored in a separate table with any relationship managed entirely in code since DDB doesn’t support joins.

In adding the DynamodDB dependencies, I found I had to update the reflection config due to a transitive dependency on Apache HTTP Client.

Kotlin

I’m learning Kotlin from scratch and it’s less intuitive than I thought, especially (ironically?) coming from Groovy. Running into issues where things don’t work and it’s not clear why. IntelliJ is a big help here, offering fixes that work and shine a light on where to look for help. I think a lot of the challenge comes from how strict the typing is. But there have also been a few times where I fired up the debugger to find out why something isn’t working… and then it just does. To be clear, it was 1) run test, which fails 2) debug run test (no changes) it now works. Like it was really an IntelliJ state problem, which I’d experienced in the past with Gradle. Refreshing the Gradle project might help, YMMV.

Kotlin has some useful features like data classes that automatically give you equals and hashcode methods, but you have to mind the constructors.