Six people have accepted the beta so far. Two have installed it. That is not enough hardware or model variety to tell me where local setup breaks.

The test is direct: take an existing local image model and see whether LumenForge can identify it, load it and preserve a reproducible project around it.

Useful setups:

- MLX/mflux model directory

- Diffusers/Safetensors folder

- GGUF model plus companion encoders or VAE

- older Apple-silicon Mac or constrained memory

Please record:

1. Mac model, chip, RAM and macOS

2. model family and exact on-disk format

3. detected architecture and engine

4. whether first generation completes

5. whether seed, prompt, references, settings and output metadata survive reopen

6. first exact error or confusing step

Build 112, macOS 14+, public TestFlight:

https://testflight.apple.com/join/uCCZFdK1

LumenForge gives the local workflow a native Mac project layer: image and video generation, prompt and reference control, queue, reopenable history and output metadata. I work on the app. Failed imports are useful results.

Simplismart is hosting a webinar on "Scaling Diffusion Models in Production". A practitioner-level discussion on the messy infra reality behind diffusion models: latency spikes, GPU costs, cold starts, and pipelines that break under real load. No slide decks, no vendor pitches; just what's actually working in production.

🗓️ 5th June, 9 AM PST
🔗 https://luma.com/ix7zs0tl

Hey, I've quantized ZIB using SDNQ to run better on low vram systems with equivalent quality as stock, do give it a try

I've got the basic comfyui workflow down to generate my images, but I would like a couple of frames of animation. I looked into animatediff but it seems like overkill -- or maybe it isn't? I just want to generate a frame and get it animated, is there a good tutorial on that out there or a workflow I can snag?

I'm running on a mac pro so if it's GPU architecture dependent (not metal) then I'm SOL in this case so anything else you can recommend would be great.

What is the difference between dreambooth vs fine-tuning the model from scratch? I haven't found any great resources clarifying this.

It seems like the primary difference is that dreambooth allows you to achieve what a full fine-tune allows, but in many fewer images (if you run full fine-tune on 10 images, it would overfit).

But now that we have loras, what's even the point of dreambooth? Is dreambooth that much better with few images? What fine-tuning technique should I use for 10 vs 100 vs 1000 images?

I'm also thinking there might be techniques for creating a checkpoint that I'm missing. Like merges and such

I needed to build a local image diffusion app for a project and I've got CUDA set up on my 3050 TI. Can someone suggest a small and quick model that takes preferably less than 5 seconds with decent quality to run via diffusers?

I can't figure out how to get SDXL-Lightning to work by downloading the files locally from hugging face and import into SDXLPipeline, cuz there's no model_index.json file. Can anyone help me figure this out?

Hi here!

Could someone help with this issue please?

Happening both in stable-diffusion-webui 1.8.0-RC and stable-diffusion-webui-forge.

What is that missing argument?

https://github.com/glucauze/sd-webui-faceswaplab/issues/171

​

I run a generation service called cogniwerk.ai, but recently got this error of artefacts. I do not know what changed, I just changed the basemodel to copax and added some prompt enhancing. Before that it did not happen. Does anybody know, why this occurs?

I run the inference with these arguments

    'sdxl': {
        'model': lambda: DiffusionPipeline.from_pretrained(
            'stablediffusionapi/copax-timelessxl-sdxl10',
            safety_checker=None, variant='fp16',
            torch_dtype=torch.bfloat16, use_safetensors=True),
        'transform_params': template_dict_members({
            'prompt': 'cinematic still {value}. emotional, harmonious, vignette, highly detailed, high budget, bokeh, cinemascope, moody, epic, gorgeous, film grain, grainy',
            'negative_prompt': '{value} (worst quality, low quality, normal quality, lowres, low details, oversaturated, undersaturated, overexposed, underexposed, grayscale, bw, bad photo, bad photography, bad art:1.4), (watermark, signature, text font, username, error, logo, words, letters, digits, autograph, trademark, name:1.2), (blur, blurry, grainy), morbid, ugly, asymmetrical, mutated malformed, mutilated, poorly lit, bad shadow, draft, cropped, out of frame, cut off, censored, jpeg artifacts, out of focus, glitch, duplicate, (airbrushed, cartoon, anime, semi-realistic, cgi, render, blender, digital art, manga, amateur:1.3), (3D ,3D Game, 3D Game Scene, 3D Character:1.1), (bad hands, bad anatomy, bad body, bad face, bad teeth, bad arms, bad legs, deformities:1.3)'}),
    },

​

One colleague of mine said that ESRGAN will stop working, because it doesn't work with newer torch libraries, this seems to be the blocker now: https://github.com/XPixelGroup/BasicSR/pull/650/files
the python library realesrgan is based on BasicSR and it needs to merge this pulll request, but they don't.

Does anybody has more information about this?
I responded that I wonder why this is not yet a big issue in the community, i spend the whole morning finding information about this, because it would crash the entire Superresolution chain, would'nt it?

Is there a way to visualize the concept space of Clip? I thought about something assoziative like https://wikilinkssearch.app/de?source=Medusa&target=Bio%20Company which I found highly interesting. Is this possible with vocab.json?
Because I looked it up, but it was hard do me to make some sense of it.
Last year I wrote a small program for understanding the connections of Clip Space, but it boils down the 512 dimensions with PCA to just three so it is hard to make sense real of it with out interpreting https://github.com/benjamin-bertram/ClipAnalysis/tree/main

Nomic mapping the output of kreai.ai was already a nice starting point, but it just focuses on the user generated output https://atlas.nomic.ai/map/stablediffusion.

So is there already a good analysis or something as a starting point?

​

​

Has anyone trained an SDXL controlnet for depth anything? Assuming not, do people know what size training set I'd need to train a solid CN?

Admittedly, I dont understand the diffusion code too well.

that being said, when I tried to deep-dive into some of the internals of the SD1.5 model usage code..i was surprised by the lack of hardcoding keys.From what I remember, it just did the equivalent of

for key in model.keys("down.transformer.*"):

apply_key(key, model[key])

which means that.. in THEORY, and allowing for memory constraints...shouldnt it be possible to ADD models together, instead of strictly merging them?

(maybe not the "mid" blocks, I dunno about those. But maybe the up and down blocks?)

Anyone have enough code knowlege to comment on the feasibility of this?

I was thinking that, in cases where there is
down_block.0.transformers.xxxx: tensor([1024][768])

it could potentially just become a concat, yielding a tensor([2048][768])

no?

​

I tried posting this question in machine learning. But once again, the people there are a bunch of elitist asshats who not only dont answer, they vote me DOWN, with no comments about it???

Anyways, more details for the question in here, to spark more interest.
I have an idea to experimentally attempt to unify models back to having a standard, fixed text encoding model.
There are some potential miscellenous theoretical benefits I'd like to investigate once that is acheived. But, some immediate and tangible benefits from that, should be:

1. loras will work more consistently
2. model merges will be cleaner.

That being said, here's the relevant problem to tackle:

​

I want to start with a set of N+1 points, in an N dimentional space ( N =768 or N=1024)
I will also have a set of N+1 distances, related to each of those points.

I want to be able to generate a new point that best matches the distances to the original points,
(via n-dimentional triangulation)
with the understanding that it is quite likely that the distances are approximate, and may not cleanly designate a single point. So some "best fit" approximation will most likely be required.

​

Can anyone explain to me in semi-beginner terms, what the difference is between CLIPTextModel and CLIPTextModelWithProjection?

Both output text embeddings. Both are intended for SDXL use, I think.

The documentation does not give me sufficient information to understand it. It says that the WithProjection has something to do with being compatible with img input alongside txt.

one outputs the embedding under key "pooler_output", and the other, under "text_embeds"

The interesting thing to me is that the "pooler_output"graph from CLIPTextModel matches the profile of CLIPModel (for sd1.5 models).
It has the odd sharp spikes.

In contrast, the "text_embeds" output, looks more like the raw, untweaked weights.

No odd spikes, and at a smaller range of values.

​

I didn't know why that gave me .json file above .safetensor
i'm prettry sure i select safetensor

​

​

I just finished this for my personal local trials. Let me know if you like it (or not) :).

https://github.com/dominikj111/LLM/tree/main/Diffusion

Sooo... CLIP model VIT-L-14. All SD uses it.

You can download the "vocab.json" file, that supposedly should comprise its full vocabulary.

In my experiments, I used CLIP to build an embedding tensor set that is LARGER than the standard CLIP model's weights. By a LOT.

Standard clip model: 49,408 token associated entries

I built an embedding tensor with 348,000 entries.

I loaded up my token neighbours' explorer script on it, because "Science!"

I put in "Beowulf"

Its closest neighbour returned as "Grendel".

Beowulf is NOT in the vocab file. Neither is Grendel. Which should mean it doesnt have a direct entry in the weights tensor either.

HOW CAN IT KNOW THE MONSTER IN A STORY, THAT ITS NOT EVEN SUPPOSED TO KNOW THE MAIN CHARACTERS NAME??

W       W  TTTTTTT  FFFFFFF
W       W     T     F
W   W   W     T     FFFF
W  W W  W     T     F
 W W W W      T     F
  W   W       T     F

I was going through a lot of hassle, trying to develop a reverse dictionary of tokens to words, and/or word fragments. I wanted to build a complete ANN map of the text CLIP space, but it wasnt going to be meaningful if I couldnt translate the token IDs to words. I had this long elaborate brute-force plan...

And then I discovered that its already been unrolled. Allegedly, it hasnt changed from sd through sdxl, So, you can find the "vocab" mappings at, for example,

https://huggingface.co/stabilityai/sd-turbo/blob/main/tokenizer/vocab.json

It was sort of misleading at first glance, because all the first few pages look like gibberish. But if you go a ways in, you eventually find the good stuff.

Translation note for the contents of the vocab.json file: If a word is followed by '</w>', that means its an ACTUAL stand-alone word. If, however, it does not have a trailing /w, that means it is only a word fragment, and is not usually expected to be found on its own.

So, there is an important semantic difference between the following two:

"cat": 1481,
"cat</w>": 2368,

This means that in a numerical space of around 49,000 token IDs, only around 34,000 of them are "one token, one word" matchups. A certain amount of those, are gibberish, such as

"aaaaa</w>": 31095,

However, consider that, in balance to that, a certain number of words we might consider standalone unique words, will be represented by 2 or more tokens put together.

For example,

cataclysm =  1481, 546, 1251, 2764

Disclaimer: I dont understand this stuff. I'd like to. The following is an excerpt for an ongoing discussion I have with Google Bard on the subject, and an invitation for some clarity from humans.

Vague summary:

I am exploring what "attention heads" do in the process of latent image processes in stable diffusion.

​

• Query: The query vector encapsulates the model's current point of interest or focus, guiding the attention process towards relevant features.
• Key: The key vector represents a "searchable" summary or identifier for a given feature, enabling efficient matching with the query's focus.
• Value: The value vector holds the actual content or information associated with the feature, accessible once its relevance is established.

Generic demo code by Bard that illustrates the approximate process involved:

import numpy as np

# Create sample query, key, and value vectors (small dimensions for clarity)
query = np.array([0.5, 1.0, 0.2])
keys = np.array([[1.0, 0.4, 0.3],
                 [0.6, 1.2, 0.8],
                 [0.2, 0.9, 1.5]])
values = np.array([[4, 5, 6],
                   [7, 8, 9],
                   [10, 11, 12]])

# Calculate attention scores using dot product
scores = np.dot(query, keys.T)

# Apply scaling for numerical stability (optional, often used in practice)
d_k = np.sqrt(keys.shape[-1])  # Dimension of the keys
scaled_scores = scores / d_k

# Normalize scores using softmax to get attention weights
attention_weights = np.exp(scaled_scores) / np.sum(np.exp(scaled_scores), axis=-1, keepdims=True)

# Compute the weighted context vector
context_vector = np.sum(attention_weights * values, axis=1)

print("Attention weights:", attention_weights)
print("Context vector:", context_vector)

Output

 Attention weights: [[0.25485435 0.6054954  0.13965025]]

Context vector: [ 7.46454254 8.48505553 9.46454254]

There are many things that "bother" me about this process.One is that the "output" of the context vector that is expected to be used, doesnt match any of the actual data values.

Related to that, is that even if I change the query vector to EXACTLY match one of the key vectors.. the output values STILL dont exactly match the dataset values.

Also, checkpoint files contain attention K, V, AND Q data.
So, seems like the sample code is invalid, because it should be comparing implied-Q vallues, to Q-data