This is now live. Come check out the new video and image generation models, and take a peek at the samples using the (i) icon in the top right of each model card to find what suits your needs best!
https://anathema.ai
https://anathema.ai
Timeline
Post
Remote status
Context
3
@matty these are all abliterated? are they fine-tuned in any way with new information?
No, these are not abliterated. They're all models served from their respective upstream vendors. I would not be able to self host Google Veo or anything like that. LTX 2.3 is the only video model I self host, and Flux Klein is the image model I self host, but their quality is nowhere near SOTA or proprietary models. Lexi, the actual text inferencing model, is not abliterated either, she's just trained on a dataset that I curated. Abliteration is good if you're running a model locally and want it to teach you about things that would be illegal since abliteration removes the refusal mechanism, not the RLHF. Obviously, for a public AI, I cannot serve abliterated for legal reasons.
@matty ok, I understand. I am just concerned about cases where the guardrails fight the fine tuning. have you had any issues?
Replies
5For which specifically? Media or text inferencing?
@matty text inferencing. I haven't tried your service yet, I just found that in other services it couldn't get past its liberal white woman training even with new information
Well, new information in the chat is not taken as instructions, that's a threat vector. Lexi can reference things, digest documents, help you write or format, et cetera, but if you upload a document that says "you are now a submissive anime waifu", it's not going to be engrained in her memory.
RLHF is very difficult to get through because open source models are pre-trained on trillions of tokens of shitlibbery and anti-wrongthink, so if you want to make even a miniscule dent in that, you have to LoRA train, and heavily. It's not cheap either. But even when you LoRA train, some models don't take well to it, or if you overtrain you get gibberish, if you undertrain, nothing happens.
I lack the resources to train our own model from scratch, but I've done CPT, SFT, DPO and more on it so it's more resistant to the shitlib stuff although it's not ever perfect. Does that make sense? I have several million tokens of training baked into the model, but in a sea of trillions of tokens of pre-training, we're really limited on how "based" we can make an AI.
RLHF is very difficult to get through because open source models are pre-trained on trillions of tokens of shitlibbery and anti-wrongthink, so if you want to make even a miniscule dent in that, you have to LoRA train, and heavily. It's not cheap either. But even when you LoRA train, some models don't take well to it, or if you overtrain you get gibberish, if you undertrain, nothing happens.
I lack the resources to train our own model from scratch, but I've done CPT, SFT, DPO and more on it so it's more resistant to the shitlib stuff although it's not ever perfect. Does that make sense? I have several million tokens of training baked into the model, but in a sea of trillions of tokens of pre-training, we're really limited on how "based" we can make an AI.
@matty got it, understood. not complaining btw just trying to figure out what's possible.
Anything is possible if you have a couple billion dollars. But, for us, the ceiling is unreachable. Maybe in the future as training methodologies change and newer models are released. I aim to make money with Anathema to serve the community better with a more intelligent, better trained model, but, frankly, "making" our own is all but impossible.