new api method for data image search

new api method for data image search

As mentioned, after doing some experiments for the KDnuggets article, I bundled some of the existing API methods into a new one, which will extract from a page figures that have x/y scaling information. The JSON output is well suited to elasticsearch or your favorite flavor of NoSQL eg., an extract of the response: [{“input”:”tmp/quant-ph0002044-9.png”,Continue Reading new api method for data image search

arxiv mining

arxiv mining

Some time ago I launched a little project, mining data from arxiv; you can read about it in other blog posts. Specifically, I modeled figures from about 500k figures as Gaussian mixture models, in order to create some features, so figures might be ultimately represented as graphs for comparison. More ordinary methods might suffice tooContinue Reading arxiv mining

API Gateway Perf

API Gateway Perf

At the time of writing, AWS API gateway doesn’t support gzip requests, so I’ve been handling this at the lambda function itself and client side. Obviously compression makes a dramatic difference w.r.t performance, just ask the guys at Pied Piper 🙂 Another curious absence is support for multipart form data; attached a screen grab fromContinue Reading API Gateway Perf

EC2 Optimization

EC2 Optimization

We attended the AWS San Francisco Summit in April and were really impressed with the breadth and depth of the presentations. A stand out included this talk from Adam Boeglin on optimization. As we prepare to finally commercialize RESTful APIs, it’s clear a few lambda functions need a little tuning; this talk as some greatContinue Reading EC2 Optimization

DevOps with AWS

DevOps with AWS

One of the distinct advantages of devOps philosophy as implemented using AWS is the ability to rapidly kick out a new service. The new template-match API is a good example. Knowing the algorithm, I was able to quickly pull something together from the p2t C++ library, an example input image attached. The algorithm determines whereContinue Reading DevOps with AWS

Soft Launch Prep

Soft Launch Prep

I’ve put some notes together, summarizing the steps involved in taking the p2t backend to AWS. Overall a very intuitive and relatively straightforward process, with very few gotchas; I cover these in these notes (recording below), and in prior blogs. As we prepare for a soft launch/ beta test, and then the collision conference inContinue Reading Soft Launch Prep

Divide & Conquer

Divide & Conquer

The day job continues to keep me busy, however, I managed to connect the dots with the compute infrastructure. Input document collections now happily cascade through a chain of processes, wrapped up as lambda functions. Meta data is pushed into dynamoDB tables, storing input and output state, allowing for browsing and downloading of outputs usingContinue Reading Divide & Conquer