This is a tale about my ongoing efforts in using AI to provide useful services. As with all things AI-related, it’ll probably be out of date within a couple of weeks of writing it — that’s just how fast things are moving these days! Hopefully it’ll give some useful info and guidance for others following a similar path of discovery!
Let’s start with a summary of the services, the hardware and the software & models.
Services
The Very Good Email Company. Using AI to assist with Phishing and Scam detection.
Open Golf Events. Using AI to process web crawls of golf websites to add events to an Open competitions database.
Edwin Chat. An AI chatbot for websites.
Hardware
2 x Asus GB10 (a variant of the DGX Spark). Chosen because I’m not willing to pay for a 4TB SSD that I don’t need when 1TB is plenty. Rack-mounted in a frame that came from Austria, of all places.
Software & Models
FreeBSD & Linux, PHP backend, NGINX web server, MariaDB, Claude Code.
Current Coding model: Opus 5.0 with a Claude Pro subscription.
Operations model: Qwen3.6–35B-A3B
Why 2 Sparks?
Let’s tackle the obvious question first. While I’d love a rack full of RTX Pro 6000 cards, it currently doesn’t make financial sense to go there. We’re based in the UK, and a ‘Spark’ box costs around £3,000. an RTX Pro 6000 is more like £12,000, plus an extra £20,000 for a server to put it in! We also want to provide a reliable service, so need at least 2 of everything…
What do we get from our Sparks? Well, roughly speaking, on a single thread with Qwen, we prefill at somewhere approaching 2,000 tokens/s, and the output runs at just under 40 tokens/s. If we ramp up to 32 concurrent sessions, we can get around 240 tokens/s, which is fine for batch processing, but no good for interactive chat.
Currently one of the boxes does the batch work, and acts as a hot spare for the other, which does the live work. Another advantage of the Sparks is the amount of memory available, which is very useful for KV cache, particularly with Edwin.chat.
Anti-Phishing
Phishing and scam emails used to be easy to spot. Mostly pretty amateurish and written by people whose first language was not English, and they didn’t have access to a spell checker! With the advent of AI, it has become a lot easier to produce well formatted emails that look like genuine ones, and thus a lot harder to spot them.
This is still very much a work in progress, but is a good illustration of the title of this piece — training the dragon. Our dragon is Qwen (Chinese dragon you see), and the training is done mostly by Claude. As you may know, to get the best out of AI you need a good harness and good prompting. This one is more about the prompt. If Qwen misclassifies something, we ask Claude to look at the email and the prompt and the classification and suggest modifications to the prompt to fix the misclassification. After many iterations over our test corpus, we can refine the prompt to increase the accuracy of the classification.
The end goal is to eventually shuffle the resulting data along to something like ModernBERT to create a much faster on-the-fly classifier as we’re processing far too many emails to run them all through the AI. But this one is still a work-in-progress as I’ve been far too distracted by the other projects…
Open Golf Events
This is a bit of a pet project. As a keen golfer, I wanted to see what could be done to produce an accurate directory/search engine for open golf competitions for amateur golfers. There is an existing service that just covers the UK, but the rest of the world was very much under-served.
Claude assisted with the coding — long before I discovered Claude Code, so it probably took about 4 times as long as it would take if I did it now! And Claude did all the AI work until I discovered it was even possible to run a model locally and save on all those token charges!
Here’s a simplified guide to how the process works.
- Crawl website, look for pages that are most likely to be lists of competitions.
- Qwen looks through the pages, and adds any competitions it sees to the database with basic details. It’s told to look for open-ness (many competitions are restricted to members only), and it may see that there’s a regular Wednesday competition, for example, so it’ll create events for Wednesdays for the next 6 months.
- Qwen is then instructed to look at each event individually and try to extract as much detail as possible. It is allowed to look at other web pages as it sees fit, and can process PDFs and even do OCR on .JPG images. Once it has that detail, it recommends either updating or removing an event, with a high, medium or low confidence. The high confidence examples tend to then be processed automatically, with some human review of the remainder.
- On a regular basis, step 3 is re-run to check for alterations, updates, cancellations etc.
All of this is run on our ‘batch’ Spark. On the server that runs these processes, we have an ai_queue table and a concurrency setting. We currently run a concurrency of 32 in the nighttime and ramp that down to 4 in the day so the box stays more responsive for other tasks.
Edwin Chat
edwin.chat is our latest project. It’s a website chatbot. Not something that I need to explain too much as a concept, but here are some of the ‘tricks’ we use to make the best use of the hardware available to us.
A chat bot doesn’t need to be a super-intelligent frontier model, so Qwen is perfectly capable of performing the job. The prompt and guardrails are very important aspects, as is caching and the joys and perils of RAG databases…
Most people have no idea how to properly prompt an AI system (and no desire to find out), so we have a database of industry-specific prompts. These ensure the AI stays on-topic, and doesn’t hallucinate to try and be too helpful. It gives it a more sales-oriented approach for Estate Agents, for example, and a more technical support oriented approach for IT Tech Support services. You get the idea… you’ll find an Edwin chatbot on each of the websites mentioned above. For a bit of fun, I created a BOFH persona which I put on this very website — feel free to go and have an argument with it!
When I started edwin.chat, I was thinking small businesses with relatively small websites. I had used a chatbot for my own website, and was not overly impressed with the results, so came up with Edwin and had Claude create a single .md file from my entire website. This was far more accurate, as it didn’t suffer from RAG database issues as everything was loaded into context.
As Edwin has developed, we’ve come across the need for RAG, and here’s the current state of affairs…
If the total data is less than 50,000 tokens, we load it all into context. First load could take 25 seconds or so, but after that, it stays in cache and is much faster (one advantage of the Spark is there’s a lot of KV Cache memory).
If it’s more than 50,000 tokens, we switch to RAG mode. Unlike many RAG systems, we don’t break at paragraph level, but at web page level, loading the 12 closest matches into context at the beginning of the conversation. These are then added to as the conversation continues, and never removed, thus avoiding Qwen losing vital context, and keeping the cache warm. If it all gets a bit too big (currently 100,000 tokens), we start again from the beginning.
I hope you’ve enjoyed the read. I’m certainly enjoying the journey and the learning. My key takeways are that good prompting and a good harness that takes advantage of the specific hardware setup are the keys to providing a good service.
Now go and train your own dragon to do something useful!