Rendered at 05:17:28 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
atmanactive 7 hours ago [-]
I wanted to start using this but I can't since I can't use any email/password combination I want. By requiring an existing account from Google/GitHub/Discord you are just contributing to corporate-owned dystopia.
yolo-auto 3 hours ago [-]
i know, but im scared to own our own auth, when there's money involved... too cheap to pay for a service. So we lean on the back of the giants for now.
If it helps, there's really no reason to ever log into our site after you get an API Key. I guess until you want to make another api key
lobster45 10 hours ago [-]
What type of hardware, GPU are you utilizing? I have the same model self hosted at home
Got me down a rabbit hole of trying to find the ROI of doing something like this. This is what every GM at AWS thinks about. So, a back of the napkin calculation of running at FP8 and approx 35GB. You can fit that on a g6e.xlarge @ 48GB for about $900/year at reserved capacity.
Assuming you are netting at $5.5 (post stripe fees), and maybe $100/month for API/auth box for egress, you probably need about 170 paying customers to breakeven. This is assuming light/bursty usage.
Kudos! This is a a win win for both you and customer using it as it removes the undifferentiated heavy lifting.
yolo-auto 3 hours ago [-]
actually the math is more like $1500 a month on a MI300x on a highly optimized rocm/sglang image and everyone's getting 100+ tok/sec and we have 100 active people and plenty of room for others.
but were losing money right now obviously.
we did come from 4x 3090s could actually make money, but we thought a couple hundred bucks a month isnt worth a 100 degree office.
People are dumping 128k cold documents in like its their job. Some dudes hit 2 billion + tokens a week.
If it helps, there's really no reason to ever log into our site after you get an API Key. I guess until you want to make another api key
~100tok/s average
Assuming you are netting at $5.5 (post stripe fees), and maybe $100/month for API/auth box for egress, you probably need about 170 paying customers to breakeven. This is assuming light/bursty usage.
Kudos! This is a a win win for both you and customer using it as it removes the undifferentiated heavy lifting.
but were losing money right now obviously.
we did come from 4x 3090s could actually make money, but we thought a couple hundred bucks a month isnt worth a 100 degree office.
People are dumping 128k cold documents in like its their job. Some dudes hit 2 billion + tokens a week.
its been fun though. you should check it out